Python [32] is a dynamically typed script programming language with a syntax inspired by functional languages such as Haskell. In this section, we will show how to use it for both DOM, SAX and StAX parsing, which we used in XML processing in Java. As most XML files use the UTF-8 encoding, we will use Version 3 of Python in which the processing of this encoding is better integrated. To make string processing more uniform, it preferable to also use UTF-8 as the encoding of our Python source file. This is indicated using a structured comment as the first line of our programs (see Example 10.6).
To show how to process an existing XML structure, we will use the compacting
process that we programmed in Java in Section 8.1. In Python, DOM
parsing is achieved simply by calling the
parse
method from the xml.dom.minidom
package. This
creates a DOM structure composed of instances of the
xml.dom.Node
class.
Example 10.6. [DOMCompact.py
] Text compaction of the cellar book
(Example 2.2) with Python using the DOM model
Compare this program with Example 8.1.
1 # -*- coding: utf-8 -*- import xml.domfrom xml.dom import Node from xml.dom.minidom import parse 5 import sys def strip_space(node):
child=node.firstChild while child!=None: 10 c=child.nextSibling if (child.nodeType==Node.TEXT_NODE and len(child.nodeValue.strip())==0): node.removeChild(child) else: 15 strip_space(child) child=c return node # use sys.stdout.write instead of print to control the output 20 # i.e. without added spaces between elements to print def compact(node,indent):
if node==None: return if node.nodeType==Node.ELEMENT_NODE:
sys.stdout.write(node.nodeName+'[') 25 indent += (len(node.nodeName)+1)*" " attrs = node.attributes first=True for i in range(len(attrs)):
if not first: sys.stdout.write('\n'+indent) 30 sys.stdout.write('@'+attrs.item(i).nodeName + '['+attrs.item(i).nodeValue+']') first=False child=node.firstChild while child!=None:
35 if not first: sys.stdout.write('\n'+indent) compact(child,indent)
first=False child=child.nextSibling sys.stdout.write(']') 40 elif node.nodeType==Node.TEXT_NODE:
sys.stdout.write(node.nodeValue.strip()) doc = parse(sys.stdin)
compact(strip_space(doc.documentElement),"") 45 sys.stdout.write('\n')
![]()
|
Imports the |
|
A recursive method which goes through the DOM structure to remove whitespace only nodes. It returns the cleaned DOM structure. |
|
Prints the content of the XML node. Each line is prefixed with an indentation, a string composed of the appropriate number of spaces. |
|
If the node is an element, prints the name of the node followed by an opening bracket and adds the appropriate number of spaces to the current indentation. |
|
Deals with attributes which are contained in a list of nodes over which we
iterate. For each attribute, prints a |
|
Loops over the list of children possibly changing line if it not the first child. |
|
Recursively calls |
|
Processes a text node by printing it and normalizing internal newlines. |
|
Parses the file by creating a new XML document from the content of the standard input. |
|
Calls the compacting process on the document node with an empty indentation string and ends the last line with a newline. |
In order to process an XML structure with the SAX approach, we will use a
similar compacting process to the one we programmed in Java in Section 8.2. SAX parsing is achieved by creating an instance of
the SAX
class to which we add a handler method for the corresponding parsing events. The
parsing process will send call-backs to the handler of these
events which will produce the compacted document.
Example 10.7. [SAXCompact.py
] Text compaction of the cellar book
(Example 2.2) with Python using the SAX model
Compare this program with Example 8.3.
1 import xml.saximport sys import CompactHandler 5 parser = xml.sax.make_parser()
parser.setContentHandler(CompactHandler.CompactHandler())
parser.parse(sys.stdin)
sys.stdout.write('\n') 10
|
Imports the SAX parser package and the |
|
Creates an instance of the SAX parser. |
|
Makes the parser send its parsing events to the
|
|
Starts the parsing process and ends the last line with a newline. |
SAX parsing in Python is only a matter of defining the appropriate handlers. Because nodes are not normalized, many successive text nodes can appear within an element, so some care has to be taken to deal with this fact.
Example 10.8. [CompactHandler.py
] Python SAX handler for text
compacting an XML file such as that of Example 2.2
Compare this program with Example 8.4.
1 from xml.sax.handler import ContentHandler import sysclass CompactHandler(ContentHandler): 5 def __init__(self):
self.closed = False self.indent = 0 self.lastNodeWasText = False 10 def startElement(self, localname, attrs):
if self.closed: sys.stdout.write('\n'+self.indent*" ") self.closed = False self.indent+=1+len(localname)
15 sys.stdout.write(localname+'[') first = True for name in attrs.getNames():
if not first: sys.stdout.write('\n'+self.indent*" ") 20 first = False sys.stdout.write('@'+name+'['+attrs.getValue(name)+']') self.closed = True self.lastNodeWasText = False 25 def endElement(self, localname):
self.closed = True sys.stdout.write(']') self.indent -= 1+len(localname) self.lastNodeWasText = False 30 def characters(self, data):
data = data.strip() if(len(data)>0): if self.closed and not self.lastNodeWasText: 35 sys.stdout.write('\n'+self.indent*" ") self.closed = False sys.stdout.write(data) self.closed=True self.lastNodeWasText = True
|
Imports the |
|
Defines the constructor of the class and initializes three instance variables needed to output the element with the correct indentation. The last flag is used when many successive text nodes appears; this happens when XML entities are being replaced by their values. |
|
When a new element is started, finishes the current indentation if needed. |
|
Updates the current indentation by adding the length of the element name. |
|
Prints the first attribute on the same line and the others on the subsequent lines properly indented. |
|
When an element finishes, closes the current bracket and updates the current indentation. |
|
For a non-empty text node, ends current line if needed and if the last node was not a text node. Writes the content of the node and indicates that the last node was a text node. |
Pull parsing in Python is performed using the pulldom
package whose
interface to the DOMEventStream
is a bit tricky to use. In particular, it
can return many successive text nodes and it does not provide an uniform API like the one
defined in the Java XMLStreamReader
class. So to simplify our processing,
we first define our own XMLStreamReader
class which will be used in Example 10.10.
Example 10.9. [XMLStreamReader.py
] Text compaction of the cellar book (Example 2.2) with Python using the StAX model
1 from xml.dom import pulldom import sysclass XMLStreamReader():
5 """ XMLStreamReader: simplified interface to DOMEventStream so that it looks similar to the Java XMLStreamReader with similar methods It concatenates all successive text nodes 10 """ def __init__(self, file):
self.events = pulldom.parse(file) self.event = None self.node = None 15 self.look_ahead=None # get next token def next(self):
if self.look_ahead == None: 20 (self.event,self.node)=self.events.getEvent() else: (self.event,self.node)=self.look_ahead self.look_ahead=None # concatenate consecutive character nodes 25 while self.event == pulldom.CHARACTERS:
self.look_ahead=self.events.__next__() while self.look_ahead[0] == pulldom.CHARACTERS: self.look_ahead=self.events.getEvent() break 30 def isWhiteSpace(self):
return (self.event == pulldom.CHARACTERS and len(self.node.data.strip())==0) 35 def isStartElement(self): return self.event == pulldom.START_ELEMENT def isEndElement(self): return self.event == pulldom.END_ELEMENT 40 def isCharacters(self): return self.event == pulldom.CHARACTERS def checkStartElement(self,method):
45 if not self.isStartElement(): raise ValueError("%s called for an event of type: %s"% (method,self.event)) def getLocalName(self):
50 self.checkStartElement("getLocalName") return self.node.localName def getAttributeCount(self): self.checkStartElement("getAttributeCount") 55 return self.node.attributes.length def getAttributeLocalName(self,i): self.checkStartElement("getAttributeLocalName") return self.node.attributes.item(i).localName 60 def getAttributeValue(self,i): self.checkStartElement("getAttributeValue") return self.node.attributes.item(i).value 65 def getText(self):
if not self.isCharacters(): raise ValueError("getText called for an event of type %s"% self.event) return self.node.data 70
|
Imports the |
|
Defines a new class to simplify access to the pull parsing process. |
|
Constructor that defines the instance variables: access to the file, the current event and node. A look-ahead node event pair useful to concatenate the text content of successive text nodes. |
|
Gets the next event either from stream if there is no look-ahead or from the look-ahead. |
|
Concatenates successive character nodes. The first non-text node is kept in the look-ahead for the next call. |
|
A series of node type tests on the current event. |
|
Checks if the current element is a start node, otherwise raises an exception. Useful for debugging the following methods. |
|
Returns information about the current start element. |
|
Checks if the current node is a text node and returns its contents if it is the case, otherwise raises an exception. |
Example 10.10. [StAXCompact.py
] Text compaction of the cellar book (Example 2.2) with Python using the StAX model
Compare this program with Example 8.5.
1 from XMLStreamReader import XMLStreamReader import sysdef compact(xmlsr,indent):
5 if xmlsr.isStartElement(): sys.stdout.write(xmlsr.getLocalName()+'[') indent += (len(xmlsr.getLocalName())+1)*" " count = xmlsr.getAttributeCount()
for i in range(count): 10 if i>0: sys.stdout.write(indent) sys.stdout.write('@'+xmlsr.getAttributeLocalName(i)+ '['+xmlsr.getAttributeValue(i)+']') first = count==0 15 while True:
xmlsr.next() while xmlsr.isWhiteSpace():xmlsr.next() if xmlsr.isEndElement():break; if first: 20 first=False else: sys.stdout.write(indent) compact(xmlsr,indent);
sys.stdout.write(']') 25 elif xmlsr.isCharacters():
sys.stdout.write(xmlsr.getText().strip()) xmlsr = XMLStreamReader(sys.stdin)
30 xmlsr.next() while not xmlsr.isStartElement():xmlsr.next()
compact(xmlsr,"\n")
sys.stdout.write("\n")
|
Imports the |
|
If it is a start element tag, outputs the name of the element followed by an opening bracket and update the current indentation. |
|
Outputs each attribute name and value properly indented except for the first one. Attributes are obtained by indexing within the loop on the number of attributes. |
|
Loops on children nodes that are not whitespace and compacts each of them with the correct indentation. |
|
Recursive call to the compacting process. |
|
Prints the normalized character content. |
|
Creates a new stream reader, indicates that it will parse the standard input. |
|
Ignores the tokens that come before the first element (e.g. processing instructions). |
|
Calls the compacting process and then prints a newline to flush the content of the last line. |
In order to demonstrate the creation of new XML document, we will parse the
compact form produced in Section 10.2.1 or in Section 10.2.2 like we did in Chapter 9.
We first need a way to access appropriate tokens corresponding
to important signals in the input file. This is achieved by defining a
CompactTokenizer
class (Example 10.11) that
will return opening and closing square brackets, at-signs and the rest as a single
string. Newlines will also delimit tokens but will be ignored in the document
processing. The file is processed by a series of calls to nextToken
or
skip
. skip
provides a useful redundancy check for
tokens that are encountered in the input but are ignored for output.
Example 10.11. [CompactTokenizer.py
] Specialized string scanner that
returns tokens of compact form.
Compare this program with Example 9.1.
1 import re, sysclass CompactTokenizer():
5 def __init__(self,file):
self.pat = re.compile('([\[@\]])') self.whitespacepat = re.compile('\n? *$') self.file = file self.tokens = [] 10 def nextToken(self):
while True: if len(self.tokens)==0: self.line = self.file.readline().strip() 15 if self.line: self.tokens = [tok for tok in
self.pat.split(self.line) if tok!=''] else: self.token = None 20 break self.token = self.tokens.pop(0) if not self.whitespacepat.match(self.token): return self.token 25 def getToken(self):
return self.token def skip(self,sym):
if self.token==sym: 30 return self.nextToken() else: raise ValueError('skip:%s expected but %s found'%(sym,self.token))
|
Includes the regular expression |
|
Defines the |
|
Defines the |
|
Gets the next token but skip those containing only whitespace. When
the current list of tokens is empty, it reads the next line of the file
and initializes the list of tokens for this line. It returns
|
|
List comprehension expression to split the current line at separators which are also returned as tokens. Should empty tokens appear, they are removed from the result. |
|
Returns the current token. |
|
Checks that the current token is the same as the one given as parameter and retrieves the next one. If the current token does not match, then raises an exception giving the expected token and the one found. |
In Python, a new document is created using the constructor of the
Document
class. The result of calling the constructor is then
assigned to a variable which can be used to create an element or a text node.
Adding a new node is done with the
appendChild
method that adds the new node as the last child of a
given node. The attributes of a node are added using the setAttribute
method.
Example 10.12. [DOMExpand.py
] Python compact form parsing to create an
XML document
A sample input for this program is Example 5.8 , which should yield Example 2.2. Compare this program with Example 9.2.
1 import xml.dom.minidom, sysfrom CompactTokenizer import CompactTokenizer
def expand(elem):
5 global ct,doc ct.skip("[") while ct.getToken()=='@':
attName = ct.nextToken() ct.nextToken() 10 elem.setAttribute(attName,ct.skip("["))
ct.nextToken() ct.skip("]") while ct.getToken()!=']':
s=ct.getToken().strip() 15 ct.nextToken() if ct.getToken()=="[": expand(elem.appendChild(doc.createElement(s.strip())))
else: elem.appendChild(doc.createTextNode(s))
20 ct.skip("]") return elem doc = xml.dom.minidom.Document()
ct = CompactTokenizer(sys.stdin)
25 while ct.nextToken()!='[':
rootName=ct.getToken() expand(doc.appendChild(doc.createElement(rootName.strip()))) print (doc.toprettyxml(" "))
30
|
Imports the necessary packages for creating DOM nodes and for accessing the standard input. |
|
Imports the |
|
Creates an XML element corresponding to the content of
|
|
Because all attribute names start with an |
|
Adding an attribute is done by calling the |
|
All children are then processed in turn until a closing square
bracket is encountered. |
|
A child node is expanded by a recursive call whose result is added as the last child of the current element. |
|
A text node is added as the last child of the current element. |
|
Initializes a new XML document. |
|
Creates the tokenizer on the standard input. |
|
The name of the root is the first token immediately followed by an
opening square bracket. The root node is created as the child of the
document node which is then filled by the initial call to
|
|
The resulting DOM node in |
Python also allows other ways of dealing with XML files. For simple reading and
manipulation of information of an XML file, one might consider the
ElementTree
package which, upon parsing an XML file, translates its
the tree structure into a Python object, an instance of the ElementTree
class, that can be processed using the usual property selectors and array iterators.
Methods are provided to get the name of an element, its attributes and its children
nodes. In an
ElementTree
instance, access to the children nodes is achieved by
iterating on the node. Text content is obtained by reading the text
property. Mixed content is dealt with the tail
property of a child node which
gives the text content in the document before the next child.
In order to illustrate the manipulation of SimpleXML structures, we give in Example 10.13 a version to produce a compact version of an XML
file. We parse (load) the file and the ElementTree
structure is built in memory. It
is then a simple matter of traversing this structure to produce a compact version of the
file.
Example 10.13. [ETCompact.py
] Python compaction of an XML file
using ElementTree
nodes.
Compaction of an XML file by first creating as ElementTree
object
and the traversing it with standard Python iterators to create a compact version of the
file.
1 import xml.etree.ElementTree as ETimport sys, re def stripNS(tag):
5 return re.sub("^\{.+\}","",tag) def compact(node,indent):
if node==None:return if ET.iselement(node): 10 localname = stripNS(node.tag)
sys.stdout.write(localname+'[') indent += (len(localname)+1)*" " attrs = node.attrib first=True 15 for name in attrs:
if not first:sys.stdout.write(indent) sys.stdout.write('@%s[%s)'%(name,attrs.get(name))) first=False if node.text and len(node.text.strip())>0:
20 sys.stdout.write(node.text.strip()) first=False for child in list(node):
if not first: sys.stdout.write(indent) compact(child,indent)
25 first=False if child.tail and len(child.tail.strip())>0: sys.stdout.write(indent+child.tail.strip()) sys.stdout.write(']') 30 compact(ET.parse(sys.stdin).getroot(),"\n")
sys.stdout.write('\n')
|
Imports the |
|
As |
|
Function to compact a node (first parameter) with each new line preceded by an indentation given by the second parameter. |
|
Prints the name of the element followed by an open bracket and updates the indentation by adding spaces corresponding to the number of characters in the element name. |
|
Prints the attributes on different lines except for the
first. The name of the attribute is preceded by |
|
If there is text content, prints it. The text is normalized by removing spaces at the start and end. |
|
If the element has children, compacts them, a new line is output if it is not the first child. |
|
Calls |
|
Parses the standard input to get the |
Example 10.14 follows the same structure as Example 10.12 to expand a compact form into a
ElementTree
structure. It uses the tokenizer of Example 10.11 to read the file. It recursively creates the
XML structure as it processes the file. As the Python library does not provide an indented form
of the XML string, we provide one here as another illustration of
ElementTree
use.
Example 10.14. [ETExpand.py
] Python compact form parsing to create a
ElementTree
document
A sample input for this program is Example 5.8 , which should yield a file equivalent to Example 2.2. Compare this program with Example 9.2.
1 import xml.etree.ElementTree as ETimport sys,re from CompactTokenizer import CompactTokenizer
5 def expand(elem): global ct ct.skip("[") while ct.getToken()=='@':
attName = ct.nextToken() 10 ct.nextToken() elem.attrib[attName]=ct.skip("[")
ct.nextToken() ct.skip("]") lastNodeWasText=True 15 while ct.getToken()!=']':
s=ct.getToken().strip() ct.nextToken() if ct.getToken()=="[":
child=ET.Element(s) 20 elem.append(child) expand(child) lastNodeWasText=False else: if lastNodeWasText:
25 if not elem.text: elem.text="" elem.text+=s else: if not child.tail: 30 child.tail="" child.tail+=s ct.skip("]") ct = CompactTokenizer(sys.stdin)
35 while ct.nextToken()!='[':
rootName=ct.getToken() doc=ET.Element(rootName.strip())
expand(doc)
40 # ElementTree does not provide a pretty-print method so we define one def pprint(elem,indent):
sys.stdout.write(indent+'<'+elem.tag) for a in elem.attrib:
sys.stdout.write(' %s="%s"'%(a,elem.attrib[a])) 45 newindent = indent+" " sys.stdout.write('>') if elem.text:
sys.stdout.write(elem.text) for child in elem:
50 pprint(child,newindent) if child.tail: sys.stdout.write(newindent+child.tail) if elem: sys.stdout.write(indent) 55 sys.stdout.write('</'+elem.tag+'>') pprint(doc,"\n")
sys.stdout.write("\n")
|
Imports the |
|
Adds the content of the file corresponding to the children of the
current node to the |
|
Because all attribute names start with an |
|
Adding an attribute is done by setting the attribute value associated
with its name in the
|
|
All children are then processed in turn until a closing square bracket
is encountered. |
|
A new element named |
|
A text node is added as the child of the current element. Successive text nodes are added by concatenating the content of the text node with the content of the last child tail. |
|
Creates the tokenizer on the standard input. |
|
The name of the root is the first token immediately followed by an opening square bracket. |
|
Creates the root node as an |
|
The root node which is filled in by a call to |
|
Recursive function to create and indented string for the current
element. The |
|
Outputs the name of element followed by its attributes on a single line. |
|
If there is a text, outputs it. |
|
Recursively process
each children possibily writing the text content of the
|
|
Prints the document node on the standard output by calling the
|