next up previous
Next: Programs Up: The BAF: A Corpus Previous: The Documents

File Formats

Right from the start, the BAF was intended to be used as an evaluation resource in the development of general-purpose alignment methods. For this reason, we were interested in ``plain-text'' documents, i.e. text files that were not tied to a specific word-processing program, and that contained no formatting or structural mark-up. In many cases, the documents were explicitly converted from proprietary formats (WordPerfect, FrameMaker, etc.) to ``plain-text'' format. In other cases, mark-up (SGML and others) was eliminated.

In its current version, the BAF corpus takes the form of a collection of computer files. Because the corpus is available in three different formats, several files correspond to each pair of documents.

1.
COAL format: This is the format in which the alignments were originally produced. A pair of documents in COAL format consists of three distinct files: two plain text files, and an alignment file. The alignment file contains a sequence of pairs [(s1,t1), (s2,t2), ..., (sn, tn)] where each pair corresponds to a segmentation point, expressed as a pair of character offsets. The interpretation of these numbers is straightforward: the segment of text in the first text file that starts at the sith character and ends just before the si+1th character corresponds to the segment of text in the second text file that extends between the tith and ti+1th characters.
2.
CES format: This SGML-based Corpus Encoding Standard was proposed jointly by Vassar College's Department of Computer Science and the Laboratoire Parole et Langage (LP&L) of The Centre National de la Recherche Scientifique (CNRS) in Aix-en-Provence, France [Ide et al., 1995]. In this format, three files also correspond to each pair of documents: two text files (CESANA format) and an alignment file (CESALIGN format). The text files are enriched with SGML mark-up that uniquely identifies each sentence in the text. The alignments are then expressed as pairs of lists of sentence identifiers.

One important difference between the COAL and CES formats is that the CES assumes a complete segmentation of the texts into sentences, which is made explicit by the mark-up. The COAL format does not make that segmentation explicit, and nothing guarantees that the segmentation that is implicit in the alignment is complete. For example, if one sentence in the English text is translated as two sentences in the French text, the boundary between the first and second French sentences will not appear in the COAL alignment.

The CES version of the BAF was produced automatically from the COAL version. While performing the conversion, we ``completed'' the implicit segmentation, by using a number of sentence-boundary detection heuristics. A random sampling of the resulting sentences reveals that the segmentation is about 97.5% correct: less than 2.5% of the sentence boundaries in the corpus are incorrect, and less than 2.5% of the real boundaries are missing.

3.
HTML format: This is a ``visualization'' format. Here, to a pair of documents corresponds a single HTML file, which can be loaded into any HTML viewer capable of displaying tables and colors.

next up previous
Next: Programs Up: The BAF: A Corpus Previous: The Documents
Michel Simard
4/24/1998