10.1. XML processing with Ruby

Ruby [81] is a dynamic object-oriented programming language known for its flexibility and uniformity when accessing object properties. Because XML tree structures can easily be represented with Ruby objects, it is possible to use a simple and intuitive way of dealing with XML objects using essentially the same syntax as that used for other Ruby objects. In Ruby, XML processing is most often performed using the REXML package. In this section, we will show how to use it for both DOM and SAX parsing, which we used in XML processing in Java.

10.1.1. DOM parsing using Ruby

To show how to process an existing XML structure, we will use the compacting process that we programmed in Java in Section 8.1. In Ruby, DOM parsing is achieved simply by creating an instance of the REXML::Document class. This instance is a node and its node type is given by its class: Element or Text in our example. Information about a node is available through properties name, attributes (a Hash whose keys are the attribute names), children (an Array that can be iterated upon with each) or value (more useful for text nodes).

Example 10.1. [DOMCompact.rb] Text compaction of the cellar book (Example 2.2) with Ruby using the DOM model

Compare this program with Example 8.1.

  1 require 'rexml/document'                                             (1)
    include REXML
    
    def compact(node,indent)                                             (2)
  5   if(node.class==Element)                                            (3)
        print node.name+"["
        indent += ' '*(node.name.length+1);
        first=true;
        node.attributes.each do |key,value|                              (4)
 10       print "\n#{indent}" unless first
          print "@#{key}[#{value}]"
          first=false;
        end
        node.children.each do |child|                                    (5)
 15       # deal only with element nodes or non-"empty" text nodes 
          if child.class==Element || child.value.strip.length>0          (6)
            print "\n#{indent}" unless first
            compact(child,indent)
            first=false;
 20       end
        end
        print "]"
      elsif node.class==Text                                             (7)
        print node.value.strip.gsub(/ *\n */," ") # normalize new lines
 25   end
    end
    
    doc = Document.new(STDIN)                                            (8)
    compact(doc.root,"")                                                 (9)
 30 

1

Makes sure that the Document class in the REXML library is present. Includes all classes of the REXML module.

2

Prints the content of the XML node. Each line prefixed with an indentation, a string composed of the appropriate number of spaces.

3

If the node is an element, prints the name of the node followed by an opening bracket and adds the appropriate number of spaces to the current indentation.

4

Deals with attributes which are contained in a hash over which we iterate. For each attribute, prints a @, its key and the corresponding value within square brackets. A new line is started for each attribute except the first.

5

Processes all children with an iterator.

6

If it is an element or a non-empty text node, recursively calls compact possibly changing line if it not the first child.

7

Processes a text node by printing it and normalizing internal newlines.

8

Parses the file by creating a new XML document from the content of the standard input.

9

Call the compacting process on the document node with an empty indentation string.


10.1.2. SAX parsing using Ruby

In order to process an XML structure with the SAX approach, we will use a similar compacting process to the one we programmed in Java in Section 8.2. SAX parsing is achieved by creating an instance of the SAX2Parser class to which we add a listener for parsing events. The parsing process will send call-backs to the handler of these events which will produce the compacted document.

Example 10.2. [SAXCompact.rb] Text compaction of the cellar book (Example 2.2) with Ruby using the SAX model

Compare this program with Example 8.3.

  1 require 'rexml/document'                                             (1)
    require 'rexml/parsers/sax2parser'
    require 'rexml/sax2listener'
    require 'CompactHandler'
  5 
    parser = REXML::Parsers::SAX2Parser.new(STDIN)                       (2)
    parser.listen(CompactHandler.new)                                    (3)
    parser.parse                                                         (4)
    

1

Makes sure that the appropriate SAX parser classes of the REXML library and the CompactHandler class are present.

2

Creates an instance of the SAX parser that will parse the standard input.

3

Has the parser send its parsing events to the CompactHandler class described in Example 10.3.

4

Starts the parsing process.


SAX parsing in Ruby is only a matter of defining the appropriate handlers. Because the current version of REXML does not handle XML entities, some code is added to deal with their declaration and their expansion.

Example 10.3. [CompactHandler.rb] Ruby SAX handler for text compacting an XML file such as that of Example 2.2

Compare this program with Example 8.4.

  1 class CompactHandler 
      include REXML::SAX2Listener                                        (1)
      def initialize                                                     (2)
        @closed=false
  5     @indent=0;
        @entities = {"&"=>"&", """=>'"', "'"=>"'", 
                     "&lt;" => "<", "&gt;" => ">"}
      end
      
 10   def start_element(uri,localname,qname,attributes)                  (3)
        if @closed
          print "\n"+" "*@indent
          @closed=false
        end
 15     @indent += 1+localname.length()                                  (4)
        print localname+"["
        first=true;
        attributes.each do |key,value|                                   (5)
          print "\n"+" "*@indent if !first
 20       first=false 
          print "@#{key}[#{expandEntities(value)}]"
          @closed=true
        end
      end
 25   
      def end_element(uri,localname,qname)                               (6)
        print "]"
        @closed=true
        @indent=@indent-1-localname.length
 30   end  
      
      def characters(text)                                               (7)
        if text.strip.length>0
          if @closed
 35         print "\n"+" "*@indent
            @closed=false
          end
          text.strip!                 # remove leading and trailing space(8)
          text.gsub!(/[\s\r\n]+/,' ') # normalize space
 40       print expandEntities(text)
          @closed=true
        end
      end
      
 45   def entitydecl(name,decl)                                          (9)
        @entities["&"+name+";"]=decl
      end
      
      def expandEntities(text)                                           (10)
 50     if text.include?("&")
          # match entities as long as there are replacements
          @entities.each{|key,value| retry if text.gsub!(key,value)}
          # replace numerical entities starting with &#
          while true
 55         if res=text.match("&#x([0-9a-fA-F]+);") # hexadecimal        (11)
              text=res.pre_match+[res[1].hex].pack("U")+res.post_match
            elsif res=text.match("&#([0-9]+);")     # decimal
              text=res.pre_match+[res[1].to_i].pack("U")+res.post_match
            else
 60           break
            end
          end
        end
        text
 65   end
    end
    

1

Includes the default declarations needed to handle events sent by a SAX parser.

2

Defines and initializes two instance variables needed to output the element with the correct indentation. Creates the entities table and initializes it with the predefined XML entities.

3

When a new element is started, finishes the current indentation if needed.

4

Updates the current indentation by adding the length of the element name.

5

Prints the first attribute on the same line and the others on the subsequent lines properly indented.

6

When an element finishes, closes the current bracket and updates the current indentation.

7

For a non-empty text node, ends current line if needed.

8

Removes leading and trailing space, normalizes and expands entities.

9

When a new entity is declared, add it to the entities table.

10

If the text contains an ampersand, it means that it contains entities the program must deal with. Entities from the entities table are expanded as often as it is necessary.

11

Numerical entities are expanded by replacing them with their Unicode equivalent.


10.1.3. Creating an XML document using Ruby

In order to demonstrate the creation of new XML document, we will parse the compact form produced in Section 10.1.1 or in Section 10.1.2 like we did in Chapter 9. We first need a way to access appropriate tokens corresponding to important signals in the input file. This is achieved by defining a CompactTokenizer class (Example 10.4) that will return opening and closing square brackets, at-signs and the rest as a single string. Newlines will also delimit tokens but will be ignored in the document processing. The file is processed by a series of calls to nextToken or skip. skip provides a useful redundancy check for tokens that are encountered in the input but are ignored for output.

Example 10.4. [CompactTokenizer.rb] Specialized string scanner that returns tokens of compact form.

Compare this program with Example 9.1.

  1 require 'strscan'                                                    (1)
    class CompactTokenizer                                               (2)
      attr_reader :token                                                 (3)
      
  5   def initialize(file)                                               (4)
        @re = /\[|\]|\n|@|[^\[\]\n@]+/ # handle different token types
        @scanner = StringScanner.new(file.read)
      end
      
 10   # get next non-whitespace token and return it
      def nextToken                                                      (5)
        loop do
          @token = @scanner.scan(@re)
          break if @token !~ /^\n? *$/ #skip tokens with only whitespace
 15     end 
        @token
      end
      
      # if the current token is "sym" then get next one 
 20   # otherwise, output error message
      def skip(sym)                                                      (6)
        if (@token==sym)
          nextToken
        else
 25       # output error message giving the position and the current line
          str = @scanner.string
          pos = @scanner.pos
          raise "skip:#{sym} expected but #{@token} found at position #{pos}:"+
                str[str.rindex("\n",pos)||0 ... str.index("\n",pos)||str.length]
 30     end
      end
    end
    

1

Includes the StringScanner package on which this tokenizer is built.

2

Defines the CompactTokenizer class.

3

Makes the current token available as a readable attribute.

4

@re is an instance variable containing the regular expression corresponding to each token type; @scanner is the StringScanner instance running on the string read from the fileName parameter.

5

Gets the next token but skip those containing only whitespace. It returns the value of the current token.

6

Checks that the current token is the same as the one given as parameter and retrieves the next one. If the current token does not match, then raises an exception giving the expected token and the one found with the content of the current line.


Thanks to the dynamic object oriented aspect of Ruby, much of the complexity of node creation and child addition is hidden in simple class instance creations: Document.new(), Element.new() and Text.new(). Adding a new node is done with the << operator that appends new information at the end of almost any object in Ruby. The attributes of a node are kept in a hash table that is an a property of an XML node. This hash table is assigned like any other one in Ruby.

Example 10.5. [DOMExpand.rb] Ruby compact form parsing to create an XML document

A sample input for this program is Example 5.8 , which should yield Example 2.2. Compare this program with Example 9.2.

  1 require 'rexml/document'                                             (1)
    include REXML
    
    require 'compactTokenizer'                                           (2)
  5 st = CompactTokenizer.new(STDIN)                                     (3)
    
    # start new document
    doc = Document.new()                                                 (4)
    doc << XMLDecl.new(1.0,"UTF-8")                                      (5)
 10 # find the name of the root
    rootName = 'dummyElement'                                            (6)
    while st.nextToken!='['
      rootName=st.token
    end
 15 
    # create the element and return it with the current token
    def expand(st,elementName)                                           (7)
      elem = Element.new(elementName)                                    (8)
      st.skip("[") 
 20   while st.token=='@' # process attributes                           (9)
        attName = st.skip("@")
        st.nextToken
        elem.attributes[attName] = st.skip("[")                          (10)
        st.nextToken
 25     st.skip("]") 
      end
      while st.token!=']' # process children                             (11)
        s = st.token.strip
        st.nextToken
 30     if(st.token=='[')
          elem << expand(st,s)                                           (12)
        else
          elem << Text.new(s)                                            (13)
        end
 35   end
      st.skip("]")
      return elem
    end
    
 40 # expand from the root element
    doc << expand(st,rootName)                                           (14)
    # write it properly indented
    doc.write(STDOUT,0)                                                  (15)
    

1

Ensures that the library REXML document class is loaded, then include it.

2

Ensures that the CompactTokenizer (Example 10.4) is loaded.

3

Creates the tokenizer on the standard input.

4

Initializes a new XML document.

5

Adds an appropriate XML declaration.

6

The name of the root is the first token immediately followed by an opening square bracket.

7

Creates an XML element corresponding to the content of rootName and returns it.

8

Creates a new, empty XML element with name elementName.

9

Because all attribute names start with an @, we loop while the current token is equal to @. The name of the attribute is saved and the following opening square bracket is skipped, the value is kept and the ] is skipped.

10

Adding an attribute is done by assigning to the attributes hash within the XML node.

11

All children are then processed in turn until a closing square bracket is encountered. s is the current token, which is either a child element name (if followed by an opening square bracket) or the content of a text node.

12

A child node is expanded by a recursive call whose result is added as the last child of the current element.

13

A text node is added as the last child of the current element.

14

Calls the expansion of the document starting from the rootName and adds the result as the child of the document.

15

Serializes the document on the standard output. By default, the XML is properly indented. The second parameter gives a global indentation level if needed.