10.2. XML processing with Python

Python [32] is a dynamically typed script programming language with a syntax inspired by functional languages such as Haskell. In this section, we will show how to use it for both DOM, SAX and StAX parsing, which we used in XML processing in Java. As most XML files use the UTF-8 encoding, we will use Version 3 of Python in which the processing of this encoding is better integrated. To make string processing more uniform, it preferable to also use UTF-8 as the encoding of our Python source file. This is indicated using a structured comment as the first line of our programs (see Example 10.6).

10.2.1. DOM parsing using Python

To show how to process an existing XML structure, we will use the compacting process that we programmed in Java in Section 8.1. In Python, DOM parsing is achieved simply by calling the parse method from the xml.dom.minidom package. This creates a DOM structure composed of instances of the xml.dom.Node class.

Example 10.6. [DOMCompact.py] Text compaction of the cellar book (Example 2.2) with Python using the DOM model

Compare this program with Example 8.1.

  1 # -*- coding: utf-8 -*-
    import xml.dom                                                       (1)
    from xml.dom import Node
    from xml.dom.minidom import parse
  5 import sys
    
    def strip_space(node):                                               (2)
        child=node.firstChild
        while child!=None:
 10         c=child.nextSibling
            if (child.nodeType==Node.TEXT_NODE and 
                len(child.nodeValue.strip())==0):
                node.removeChild(child)
            else:
 15             strip_space(child)
            child=c
        return node
    
    #  use sys.stdout.write instead of print to control the output
 20 #  i.e. without added spaces between elements to print
    def compact(node,indent):                                            (3)
        if node==None: return
        if node.nodeType==Node.ELEMENT_NODE:                             (4)
            sys.stdout.write(node.nodeName+'[')
 25         indent += (len(node.nodeName)+1)*" "
            attrs = node.attributes
            first=True
            for i in range(len(attrs)):                                  (5)
                if not first: sys.stdout.write('\n'+indent)
 30             sys.stdout.write('@'+attrs.item(i).nodeName + 
                                 '['+attrs.item(i).nodeValue+']')
                first=False
            child=node.firstChild
            while child!=None:                                           (6)
 35             if not first: sys.stdout.write('\n'+indent)
                compact(child,indent)                                    (7)
                first=False
                child=child.nextSibling
            sys.stdout.write(']')
 40     elif node.nodeType==Node.TEXT_NODE:                              (8)
            sys.stdout.write(node.nodeValue.strip())
    
    doc = parse(sys.stdin)                                               (9)
    compact(strip_space(doc.documentElement),"")
 45 sys.stdout.write('\n')                                               (10)
    
    
    

1

Imports the xml.dom module from which we specifically import the Node class and the parse method to read the standard input.

2

A recursive method which goes through the DOM structure to remove whitespace only nodes. It returns the cleaned DOM structure.

3

Prints the content of the XML node. Each line is prefixed with an indentation, a string composed of the appropriate number of spaces.

4

If the node is an element, prints the name of the node followed by an opening bracket and adds the appropriate number of spaces to the current indentation.

5

Deals with attributes which are contained in a list of nodes over which we iterate. For each attribute, prints a @, its name and the corresponding value within square brackets. A new line is started for each attribute except the first.

6

Loops over the list of children possibly changing line if it not the first child.

7

Recursively calls compact on the child node with the updated indentation.

8

Processes a text node by printing it and normalizing internal newlines.

9

Parses the file by creating a new XML document from the content of the standard input.

10

Calls the compacting process on the document node with an empty indentation string and ends the last line with a newline.


10.2.2. SAX parsing using Python

In order to process an XML structure with the SAX approach, we will use a similar compacting process to the one we programmed in Java in Section 8.2. SAX parsing is achieved by creating an instance of the SAX class to which we add a handler method for the corresponding parsing events. The parsing process will send call-backs to the handler of these events which will produce the compacted document.

Example 10.7. [SAXCompact.py] Text compaction of the cellar book (Example 2.2) with Python using the SAX model

Compare this program with Example 8.3.

  1 import xml.sax                                                       (1)
    import sys
    import CompactHandler 
    
  5 parser = xml.sax.make_parser()                                       (2)
    parser.setContentHandler(CompactHandler.CompactHandler())            (3)
    parser.parse(sys.stdin)                                              (4)
    sys.stdout.write('\n')
    
 10 

1

Imports the SAX parser package and the CompactHandler class.

2

Creates an instance of the SAX parser.

3

Makes the parser send its parsing events to the CompactHandler class described in Example 10.8.

4

Starts the parsing process and ends the last line with a newline.


SAX parsing in Python is only a matter of defining the appropriate handlers. Because nodes are not normalized, many successive text nodes can appear within an element, so some care has to be taken to deal with this fact.

Example 10.8. [CompactHandler.py] Python SAX handler for text compacting an XML file such as that of Example 2.2

Compare this program with Example 8.4.

  1 from xml.sax.handler import ContentHandler
    import sys                                                           (1)
    
    class CompactHandler(ContentHandler):
  5     def __init__(self):                                              (2)
            self.closed = False
            self.indent = 0
            self.lastNodeWasText = False
     
 10     def startElement(self, localname, attrs):                        (3)
            if self.closed:
                sys.stdout.write('\n'+self.indent*" ")
                self.closed = False
            self.indent+=1+len(localname)                                (4)
 15         sys.stdout.write(localname+'[')
            first = True
            for name in attrs.getNames():                                (5)
                if not first: 
                    sys.stdout.write('\n'+self.indent*" ")
 20             first = False
                sys.stdout.write('@'+name+'['+attrs.getValue(name)+']')
                self.closed = True
            self.lastNodeWasText = False
     
 25     def endElement(self, localname):                                 (6)
            self.closed = True
            sys.stdout.write(']')
            self.indent -= 1+len(localname)
            self.lastNodeWasText = False
 30 
        def characters(self, data):                                      (7)
            data = data.strip()
            if(len(data)>0): 
                if self.closed and not self.lastNodeWasText:
 35                 sys.stdout.write('\n'+self.indent*" ")
                    self.closed = False
                sys.stdout.write(data)
                self.closed=True
            self.lastNodeWasText = True

1

Imports the xml.sax.handler package containing the ContentHandler class which will be subclassed to handle events sent by a SAX parser.

2

Defines the constructor of the class and initializes three instance variables needed to output the element with the correct indentation. The last flag is used when many successive text nodes appears; this happens when XML entities are being replaced by their values.

3

When a new element is started, finishes the current indentation if needed.

4

Updates the current indentation by adding the length of the element name.

5

Prints the first attribute on the same line and the others on the subsequent lines properly indented.

6

When an element finishes, closes the current bracket and updates the current indentation.

7

For a non-empty text node, ends current line if needed and if the last node was not a text node. Writes the content of the node and indicates that the last node was a text node.


10.2.3. StAX parsing using Python

Pull parsing in Python is performed using the pulldom package whose interface to the DOMEventStream is a bit tricky to use. In particular, it can return many successive text nodes and it does not provide an uniform API like the one defined in the Java XMLStreamReader class. So to simplify our processing, we first define our own XMLStreamReader class which will be used in Example 10.10.

Example 10.9. [XMLStreamReader.py] Text compaction of the cellar book (Example 2.2) with Python using the StAX model

  1 from xml.dom import pulldom
    import sys                                                           (1)
    
    class XMLStreamReader():                                             (2)
  5     """
    XMLStreamReader: simplified interface to DOMEventStream
    so that it looks similar to the Java XMLStreamReader with similar methods
    It concatenates all successive text nodes
    
 10 """
        def __init__(self, file):                                        (3)
            self.events = pulldom.parse(file)
            self.event = None
            self.node  = None
 15         self.look_ahead=None
    
        # get next token 
        def next(self):                                                  (4)
            if self.look_ahead == None:
 20             (self.event,self.node)=self.events.getEvent()
            else:
                (self.event,self.node)=self.look_ahead
                self.look_ahead=None
            # concatenate consecutive character nodes 
 25         while self.event == pulldom.CHARACTERS:                      (5)
                self.look_ahead=self.events.__next__()
                while self.look_ahead[0] == pulldom.CHARACTERS:
                    self.look_ahead=self.events.getEvent()
                break
 30                                         
        def isWhiteSpace(self):                                          (6)
            return (self.event == pulldom.CHARACTERS and 
                    len(self.node.data.strip())==0)
    
 35     def isStartElement(self):
            return self.event == pulldom.START_ELEMENT
    
        def isEndElement(self):
            return self.event == pulldom.END_ELEMENT
 40 
        def isCharacters(self):
            return self.event == pulldom.CHARACTERS
    
        def checkStartElement(self,method):                              (7)
 45         if not self.isStartElement():
                raise ValueError("%s called for an event of type: %s"%
                                  (method,self.event))
        
        def getLocalName(self):                                          (8)
 50         self.checkStartElement("getLocalName")
            return self.node.localName
    
        def getAttributeCount(self):
            self.checkStartElement("getAttributeCount")
 55         return self.node.attributes.length
    
        def getAttributeLocalName(self,i):
            self.checkStartElement("getAttributeLocalName")
            return self.node.attributes.item(i).localName
 60 
        def getAttributeValue(self,i):
            self.checkStartElement("getAttributeValue")
            return self.node.attributes.item(i).value
    
 65     def getText(self):                                               (9)
            if not self.isCharacters():
                raise ValueError("getText called for an event of type %s"%
                                  self.event)
            return self.node.data
 70 

1

Imports the pulldom package.

2

Defines a new class to simplify access to the pull parsing process.

3

Constructor that defines the instance variables: access to the file, the current event and node. A look-ahead node event pair useful to concatenate the text content of successive text nodes.

4

Gets the next event either from stream if there is no look-ahead or from the look-ahead.

5

Concatenates successive character nodes. The first non-text node is kept in the look-ahead for the next call.

6

A series of node type tests on the current event.

7

Checks if the current element is a start node, otherwise raises an exception. Useful for debugging the following methods.

8

Returns information about the current start element.

9

Checks if the current node is a text node and returns its contents if it is the case, otherwise raises an exception.


Example 10.10. [StAXCompact.py] Text compaction of the cellar book (Example 2.2) with Python using the StAX model

Compare this program with Example 8.5.

  1 from XMLStreamReader import XMLStreamReader
    import sys                                                           (1)
            
    def compact(xmlsr,indent):                                           (2)
  5     if xmlsr.isStartElement():
            sys.stdout.write(xmlsr.getLocalName()+'[')
            indent += (len(xmlsr.getLocalName())+1)*" "
            count = xmlsr.getAttributeCount()                            (3)
            for i in range(count):
 10             if i>0: 
                    sys.stdout.write(indent)
                sys.stdout.write('@'+xmlsr.getAttributeLocalName(i)+
                                 '['+xmlsr.getAttributeValue(i)+']')
            first = count==0
 15         while True:                                                  (4)
                xmlsr.next()
                while xmlsr.isWhiteSpace():xmlsr.next()
                if xmlsr.isEndElement():break;
                if first: 
 20                 first=False
                else:
                    sys.stdout.write(indent)
                compact(xmlsr,indent);                                   (5)
            sys.stdout.write(']')
 25     elif xmlsr.isCharacters():                                       (6)
            sys.stdout.write(xmlsr.getText().strip())
                
    
    xmlsr = XMLStreamReader(sys.stdin)                                   (7)
 30 xmlsr.next()
    while not xmlsr.isStartElement():xmlsr.next()                        (8)
    compact(xmlsr,"\n")                                                  (9)
    sys.stdout.write("\n")
    

1

Imports the XMLStreamReader class defined in Example 10.9

2

If it is a start element tag, outputs the name of the element followed by an opening bracket and update the current indentation.

3

Outputs each attribute name and value properly indented except for the first one. Attributes are obtained by indexing within the loop on the number of attributes.

4

Loops on children nodes that are not whitespace and compacts each of them with the correct indentation.

5

Recursive call to the compacting process.

6

Prints the normalized character content.

7

Creates a new stream reader, indicates that it will parse the standard input.

8

Ignores the tokens that come before the first element (e.g. processing instructions).

9

Calls the compacting process and then prints a newline to flush the content of the last line.


10.2.4. Creating an XML document using Python

In order to demonstrate the creation of new XML document, we will parse the compact form produced in Section 10.2.1 or in Section 10.2.2 like we did in Chapter 9. We first need a way to access appropriate tokens corresponding to important signals in the input file. This is achieved by defining a CompactTokenizer class (Example 10.11) that will return opening and closing square brackets, at-signs and the rest as a single string. Newlines will also delimit tokens but will be ignored in the document processing. The file is processed by a series of calls to nextToken or skip. skip provides a useful redundancy check for tokens that are encountered in the input but are ignored for output.

Example 10.11. [CompactTokenizer.py] Specialized string scanner that returns tokens of compact form.

Compare this program with Example 9.1.

  1 import re, sys                                                       (1)
    
    class CompactTokenizer():                                            (2)
        
  5     def __init__(self,file):                                         (3)
            self.pat = re.compile('([\[@\]])')
            self.whitespacepat = re.compile('\n? *$')
            self.file = file
            self.tokens = []
 10 
        def nextToken(self):                                             (4)
            while True:
                if len(self.tokens)==0:
                    self.line = self.file.readline().strip()
 15                 if self.line:
                        self.tokens = [tok for tok in                    (5)
                                       self.pat.split(self.line) if tok!='']
                    else:
                        self.token = None
 20                     break
                self.token = self.tokens.pop(0)
                if not self.whitespacepat.match(self.token):
                    return self.token
                
 25     def getToken(self):                                              (6)
            return self.token
    
        def skip(self,sym):                                              (7)
            if self.token==sym:
 30             return self.nextToken()
            else:
                raise ValueError('skip:%s expected but %s found'%(sym,self.token))

1

Includes the regular expression re package on which this tokenizer is built.

2

Defines the CompactTokenizer class.

3

Defines the pat regular expression to split an input line on an ampersand, an opening or closing square bracket. The square brackets characters in the regular expression must be preceded by a backslash as square brackets are part of the regular expression language, also used in the same expression. Defines a regular expression to match a whitespace node. Save the reference to the file to tokenize and initializes the current list of tokens.

4

Gets the next token but skip those containing only whitespace. When the current list of tokens is empty, it reads the next line of the file and initializes the list of tokens for this line. It returns None at the end of the file. When the list of tokens is not empty, sets the current token to the first one in the list and removes it from the list. If the current token is not a whitespace node, returns it.

5

List comprehension expression to split the current line at separators which are also returned as tokens. Should empty tokens appear, they are removed from the result.

6

Returns the current token.

7

Checks that the current token is the same as the one given as parameter and retrieves the next one. If the current token does not match, then raises an exception giving the expected token and the one found.


In Python, a new document is created using the constructor of the Document class. The result of calling the constructor is then assigned to a variable which can be used to create an element or a text node. Adding a new node is done with the appendChild method that adds the new node as the last child of a given node. The attributes of a node are added using the setAttribute method.

Example 10.12. [DOMExpand.py] Python compact form parsing to create an XML document

A sample input for this program is Example 5.8 , which should yield Example 2.2. Compare this program with Example 9.2.

  1 import xml.dom.minidom, sys                                          (1)
    from CompactTokenizer import CompactTokenizer                        (2)
    
    def expand(elem):                                                    (3)
  5     global ct,doc
        ct.skip("[")
        while ct.getToken()=='@':                                        (4)
            attName = ct.nextToken()
            ct.nextToken()
 10         elem.setAttribute(attName,ct.skip("["))                      (5)
            ct.nextToken()
            ct.skip("]")
        while ct.getToken()!=']':                                        (6)
            s=ct.getToken().strip()
 15         ct.nextToken()
            if ct.getToken()=="[":
                expand(elem.appendChild(doc.createElement(s.strip())))   (7)
            else:
                elem.appendChild(doc.createTextNode(s))                  (8)
 20     ct.skip("]")
        return elem
    
    doc = xml.dom.minidom.Document()                                     (9)
    ct = CompactTokenizer(sys.stdin)                                     (10)
 25 while ct.nextToken()!='[':                                           (11)
        rootName=ct.getToken()
    expand(doc.appendChild(doc.createElement(rootName.strip())))
    
    print (doc.toprettyxml("  "))                                        (12)
 30 
    

1

Imports the necessary packages for creating DOM nodes and for accessing the standard input.

2

Imports the CompactTokenizer (Example 10.11).

3

Creates an XML element corresponding to the content of elem and returns it.

4

Because all attribute names start with an @, we loop while the current token is equal to @. The name of the attribute is saved and the following opening square bracket is skipped, the value is kept and the ] is skipped.

5

Adding an attribute is done by calling the setAttribute method with the name of the attribute and its corresponding value.

6

All children are then processed in turn until a closing square bracket is encountered. s is the current token, which is either a child element name (if followed by an opening square bracket) or the content of a text node.

7

A child node is expanded by a recursive call whose result is added as the last child of the current element.

8

A text node is added as the last child of the current element.

9

Initializes a new XML document.

10

Creates the tokenizer on the standard input.

11

The name of the root is the first token immediately followed by an opening square bracket. The root node is created as the child of the document node which is then filled by the initial call to expand.

12

The resulting DOM node in doc is then formatted in indented form to the output using the toprettyxml method.


10.2.5. Other means of dealing with XML documents using Python

Python also allows other ways of dealing with XML files. For simple reading and manipulation of information of an XML file, one might consider the ElementTree package which, upon parsing an XML file, translates its the tree structure into a Python object, an instance of the ElementTree class, that can be processed using the usual property selectors and array iterators. Methods are provided to get the name of an element, its attributes and its children nodes. In an ElementTree instance, access to the children nodes is achieved by iterating on the node. Text content is obtained by reading the text property. Mixed content is dealt with the tail property of a child node which gives the text content in the document before the next child.

In order to illustrate the manipulation of SimpleXML structures, we give in Example 10.13 a version to produce a compact version of an XML file. We parse (load) the file and the ElementTree structure is built in memory. It is then a simple matter of traversing this structure to produce a compact version of the file.

Example 10.13. [ETCompact.py] Python compaction of an XML file using ElementTree nodes.

Compaction of an XML file by first creating as ElementTree object and the traversing it with standard Python iterators to create a compact version of the file.

  1 import xml.etree.ElementTree as ET                                   (1)
    import sys, re
    
    def stripNS(tag):                                                    (2)
  5     return re.sub("^\{.+\}","",tag)
    
    def compact(node,indent):                                            (3)
        if node==None:return
        if ET.iselement(node):
 10         localname = stripNS(node.tag)                                (4)
            sys.stdout.write(localname+'[')
            indent += (len(localname)+1)*" "
            attrs = node.attrib
            first=True
 15         for name in attrs:                                           (5)
                if not first:sys.stdout.write(indent)
                sys.stdout.write('@%s[%s)'%(name,attrs.get(name)))
                first=False
            if node.text and len(node.text.strip())>0:                   (6)
 20             sys.stdout.write(node.text.strip())
                first=False
            for child in list(node):                                     (7)
                if not first: sys.stdout.write(indent)
                compact(child,indent)                                    (8)
 25             first=False
                if child.tail and len(child.tail.strip())>0:
                    sys.stdout.write(indent+child.tail.strip())
            sys.stdout.write(']')
    
 30 compact(ET.parse(sys.stdin).getroot(),"\n")                          (9)
    sys.stdout.write('\n')
    

1

Imports the ElementTree class and renames is as ET because it will be used often. Imports other necessary packages.

2

As ElementTree nodes contain namespace information in their print names, we discard it here to keep only the local name.

3

Function to compact a node (first parameter) with each new line preceded by an indentation given by the second parameter.

4

Prints the name of the element followed by an open bracket and updates the indentation by adding spaces corresponding to the number of characters in the element name.

5

Prints the attributes on different lines except for the first. The name of the attribute is preceded by @ and the value is put within square brackets.

6

If there is text content, prints it. The text is normalized by removing spaces at the start and end.

7

If the element has children, compacts them, a new line is output if it is not the first child.

8

Calls compact recursively.

9

Parses the standard input to get the ElementTree root node and calls the compacting process on it. Ends the output with a new line.


Example 10.14 follows the same structure as Example 10.12 to expand a compact form into a ElementTree structure. It uses the tokenizer of Example 10.11 to read the file. It recursively creates the XML structure as it processes the file. As the Python library does not provide an indented form of the XML string, we provide one here as another illustration of ElementTree use.

Example 10.14. [ETExpand.py] Python compact form parsing to create a ElementTree document

A sample input for this program is Example 5.8 , which should yield a file equivalent to Example 2.2. Compare this program with Example 9.2.

  1 import xml.etree.ElementTree as ET                                   (1)
    import sys,re
    from CompactTokenizer import CompactTokenizer
                                                                         (2)
  5 def expand(elem):
        global ct
        ct.skip("[")
        while ct.getToken()=='@':                                        (3)
            attName = ct.nextToken()
 10         ct.nextToken()
            elem.attrib[attName]=ct.skip("[")                            (4)
            ct.nextToken()
            ct.skip("]")
        lastNodeWasText=True
 15     while ct.getToken()!=']':                                        (5)
            s=ct.getToken().strip()
            ct.nextToken()
            if ct.getToken()=="[":                                       (6)
                child=ET.Element(s)
 20             elem.append(child)
                expand(child)
                lastNodeWasText=False
            else:
                if lastNodeWasText:                                      (7)
 25                 if not elem.text:
                        elem.text=""
                    elem.text+=s
                else:
                    if not child.tail:
 30                     child.tail=""
                    child.tail+=s
        ct.skip("]")
    
    ct = CompactTokenizer(sys.stdin)                                     (8)
 35 while ct.nextToken()!='[':                                           (9)
        rootName=ct.getToken()
    doc=ET.Element(rootName.strip())                                     (10)
    expand(doc)                                                          (11)
    
 40 # ElementTree does not provide a pretty-print method so we define one
    def pprint(elem,indent):                                             (12)
        sys.stdout.write(indent+'<'+elem.tag)    
        for a in elem.attrib:                                            (13)
            sys.stdout.write(' %s="%s"'%(a,elem.attrib[a]))
 45     newindent = indent+"   "
        sys.stdout.write('>')
        if elem.text:                                                    (14)
            sys.stdout.write(elem.text)
        for child in elem:                                               (15)
 50         pprint(child,newindent)
            if child.tail:
                sys.stdout.write(newindent+child.tail)
        if elem:
            sys.stdout.write(indent)
 55     sys.stdout.write('</'+elem.tag+'>')
    
    pprint(doc,"\n")                                                     (16)
    sys.stdout.write("\n")

1

Imports the ElementTree, other system packages and the CompactTokenizer (Example 10.11).

2

Adds the content of the file corresponding to the children of the current node to the elem element.

3

Because all attribute names start with an @, loops while the current token is equal to @. The name of the attribute is saved and the following opening square bracket is skipped, the value is kept and the ] is skipped.

4

Adding an attribute is done by setting the attribute value associated with its name in the attrib dictionary of the current element.

5

All children are then processed in turn until a closing square bracket is encountered. s is the current token, which is either a child element name (if followed by an opening square bracket) or the content of a text node.

6

A new element named s is created and its children are filled in by a recursive call to expand.

7

A text node is added as the child of the current element. Successive text nodes are added by concatenating the content of the text node with the content of the last child tail.

8

Creates the tokenizer on the standard input.

9

The name of the root is the first token immediately followed by an opening square bracket.

10

Creates the root node as an ElementTree.

11

The root node which is filled in by a call to expand.

12

Recursive function to create and indented string for the current element. The indent parameter is a string which is output before the start of each new line.

13

Outputs the name of element followed by its attributes on a single line.

14

If there is a text, outputs it.

15

Recursively process each children possibily writing the text content of the tail property at the same level of indentation. Finally closes the tag.

16

Prints the document node on the standard output by calling the pprint method defined above.