ExamineXML: Exploring and Extracting Nodes from Large XML Files

Context of use

The size of XML files can grow fast (in some cases, they can be many megabytes long) and thus complicate the access to their content (i.e. identification and output of a few nodes). XML file exploration and extraction tools rely on XPath processors that evaluate a tree-oriented expression over the global tree structure to select nodes in the tree. In order to evaluate such an expression, the XPath processor first reads the file and builds, in memory, an internal tree structure over which the expression is evaluated. This is not a problem with small files (less than a megabyte or two) but it becomes problematic for larger ones because, in memory, the tree structure is much bigger than the original text file.

For extracting small portions of large text files, we often resort to sequential reading of each line and decide for each if it is to be kept or not. A well-known utility for this operation is grep which reads a file line by line and copies only the ones matching a given regular expression. In the case of tree-oriented XML files, grep is not very useful because the regular expression cannot take into account the embedded structure of the tags. And worse, in many cases, XML files are composed of a single line (to save space...) because end-of-lines do not have any special meaning except within character content. It is the tree structure that matters and it must be taken into account in the extraction process. Our goal is to find a utility similar to grep but operating on big XML files.

We suggest the equivalent of a line in a tree-structured XML file in order to be able to process the file sequentially by evaluating a somewhat restricted, but nevertheless useful, form of XPath expression in order to select some parts of an XML without having to build the tree in memory before evaluation.

The implementation is a Java 1.6 program that reads the XML file and prints the XPath expression leading to each node or selects nodes matching a given XPath expression. The motivation and underlying algorithm is described in this document.

Getting the program

Calling the program

CAVEAT

Because of a bug in the JDK 1.6, an out of memory error can be raised when dealing with files referencing a DTD. Should you get this type of error, try removing the DTD or use the SAX parser with the -sax option on the command line.

Options for both exploration and extraction

-h print a reminder of the usage of the program on standard output and exit
-d print debugging information on the standard output
-st print program execution statistics on the standard error
-o name of the output file for the results, standard output if none is specified
-sax use the SAX parser instead of the pull parser

Exploring the XML file

This mode is used when no -xp option is present. This scans the input file and prints either all XPath expressions leading to nodes as they are encountered. Alternatively a table with the frequency of each XPath expression can be obtained.

Options for the exploration

-a show attributes for elements
-av show attribute values for elements (implies -a)
-v show values of text nodes
-sa sort xpaths alphabetically
-sf sort xpaths by decreasing frequency
-ns display namespaces at the end of the table

Extracting nodes from the XML file

This mode is selected when the -xp option is present. This scans the file and prints each node matching the XPath expression following the -xp argument. When called from the shell command line, some care has to be taken to avoid interpretation of some special characters such as brackets, parentheses and quote. Most of the time, putting the expression within double quotes is sufficient. It is possible to use the -xpt argument to print the XPath expression as it is seen by the program and exit.

Options for extraction

-xp followed by an XPath expression of nodes to output, whitespace nodes are ignored the expression must return a NODESET and should not make any 'outside' references
-xpt followed by an XPath expression, prints the XPath expression as seen by the program and exits. Useful to check that the shell did not interpret some special characters
-xsl name of XSLT transformation file with templates to apply to each result node; by default this uses XSL 1.0 unless the Saxon.jar is added to the class path at the call
-xslf name of XSLT transformation file with templates to apply to an XML document (this document is built in memory) containing the extracted nodes; by default this uses XSL 1.0 unless the Saxon.jar is added to the class path at the call
-r name of root element, 'root' if not specified
-nr do not output root node
-c output only count of output nodes on standard error
-pp pretty print the XML result nodes
-max next arg is a positive integer giving the maximum number or results needed
-xmlns: followed immediately by pref=URI is a prefix of a namespace to add to the root (can be repeated)

Sample use cases on a small XML file

Given this Figure1.xml file:
<r>
    <a id="i1"/>
    <b id="i2">
        <c id="i3">e</c>d<c id="i4">f</c>
    </b>
</r>

which corresponds to this tree:

Here are some same calls and results:

Implementation notes

What makes this program memory efficient is the use of Pull Parser used to read each XML tag and content as needed and the construction of a skeleton document, a DOM document containing only the nodes between the root and a leaf of the full document. As each node of the full document is read, the skeleton is updated to keep only the path between the node and the root. For exploration, this path is printed and, for extraction, the XPath expression is evaluated on this skeleton document.

In order for the evaluation of an XPath expression over the skeleton document to give the same results as its evaluation over the whole document, the allowed XPath expression language is limited somewhat with the following restrictions:

In practice, for the purpose of extracting nodes from an XML file, these limitations are not very problematic. The language of extraction is thus similar to the regular expression language used by tools such as grep. In our implementation, each extracted node is further transformed with an XSL stylesheet (which is the identity transformation if none is specified) in which the full XPath transformation language is available.

More details are available in this document

You can also have a look at the source file directory is here and the Javadoc directory or get the two as a Zip file