Translated from:ParsCit: An open-source CRF reference string parsing package (partial)

0 Abstract

We describe ParsCit: a free, open-source reference parsing package. At its core, ParsCit uses a trained CRF model to label sequences of tokens in reference strings. This heuristic model segments and recognizes plain-text files. The package can be run standalone or used as a web server.

We compare ParsCit using three different datasets and benchmark it against previously published work.

1 Introduction

In scholarly works, we acknowledge prior contributions through references. The final section of a paper typically lists cited works as a reference list or bibliography. This form of acknowledgment is essential in helping readers and reviewers connect the present work with preceding and subsequent research discussions.

One project actively pursued in the bibliographic research community is automatically constructing citation networks from underlying source documents. A prerequisite is to programmatically restore the links between citing and cited documents, which requires a machine to understand the relationships encoded in reference strings. Each reference string can be viewed as a set of fields (e.g., author, title, year, journal) expressed as a string, whose content is recovered using punctuation and similar cues. However, parsing these reference strings is difficult to automate because of human conventions, varying standards across communities, and occasional author errors or oversights.

Many methods for addressing this sequence-labeling task have been proposed (Peng and McCallum, 2004; Giles et al., 1998). In this paper, we describe ParsCit, a framework and system that uses machine learning and heuristic processing.

Although many methods have been proposed for this specific problem (Huang et al., 2004; Cortez et al., 2007), our contributions are: (1) new features designed for this task, (2) automatic extraction of citation contexts, (3) packaging our results as a software module that can serve as a foundational service or as a web server, and (4) releasing our code as open source for the benefit of the community.

In the remainder of this paper, we discuss the core machine-learning model, followed by detailed preprocessing and postprocessing steps for a usable service. We then describe implementation details and the toolkits used, and summarize and compare related work.

2 Learning Model

First, we formally define the problem. A reference string R is initially decomposed into a sequence of tokens {r1, r2, ... rn}. Each token is assigned a correct class label from C = {c1, c2, ... cn}. This formulation can naturally use any data derived from citation strings, as well as any previously assigned classifications among r1 ... rn.

Sequence labeling is a common task in natural language processing, including part-of-speech tagging, chunking, and semantic role labeling. It also suggests that solutions from such tasks can inform this problem. That is, metadata are labeled as a sequence of classes such as author, title, journal, etc. In our implementation, a total of 13 classes are used, corresponding to those commonly found in reference-management software (e.g., EndNote, BibTeX).

We learn a model in the form of conditional random fields (Lafferty et al., 2001), which can be applied to unseen data. This learning model is easy to evaluate and handles large datasets. We list the features used by the ParsCit system in general; the numbers in parentheses indicate the number of feature modules in each category.

  1. Token identity (3): We have three modules that encode features in the following three forms: (1) as-is, (2) lowercased, and (3) lowercased without punctuation.
  2. N-gram prefixes/suffixes (9): We have four features that encode the first four characters of a token and, likewise, four features that encode the last four characters. An additional feature checks the last character of the token to see whether it is uppercase, lowercase, or a digit.
  3. Spelling form (1): We analyze the case pattern of the token and assign it one of four values: initial-capitalized, mixed-case, all-lowercase, or other.
  4. Punctuation (1): Similarly, we provide a detailed classification of punctuation in the token, such as: opening quote, closing quote, multiple hyphens (occasionally found in page ranges), continuation punctuation (e.g., comma, semicolon), stop punctuation (e.g., period, double quote), paired parentheses, possible volume label (e.g., "3(4)"), or other.
  5. Numerical values (1): We analyze tokens that may contain numbers. The values in this module are specific, such as year (a value between 19xx and 20xx), possible page range (including [0-9]([0-9]*)), possible volume label (including [0-9]([0-9]*)), ordinal numbers (including those ending in "th"), or general categories: four-digit, three-digit, two-digit, one-digit, contains digits, and contains no digits.
  6. Dictionaries (6): Separate analyses check whether a token matches a known publisher name, place name, surname, female name, or male name; these values are stored in a hash table.
  7. Position (1): We encode the token's position by discretizing it into n equal-width bins (through experiments, we set n to 12). In most cases, important values such as author, title, and year tend to appear toward the beginning; this feature helps the CRF learn the sequential structure of tokens.
  8. Probable editors (1): This feature indicates whether a preceding keyword such as "eds." is present.

Note that many of our features (e.g., spelling form, numerical values, punctuation) make the distinctions much finer-grained, which greatly improves performance over previous work. At the same time, we observe that authors and editors were often incorrectly separated in earlier work; to correct this, we explicitly add the editor feature so that the two are more distinguishable and easier to separate correctly.

3 Preprocessing

Before reference strings can be correctly extracted, we first need to locate the references section of an article. Although formatting (e.g., font changes) can help distinguish it, relying on specific formatting reduces generality. For this reason, ParsCit assumes that documents are first converted to plain text encoded in UTF-8. Extracting from highly formatted document types such as PDF is therefore very difficult, but proper extraction is necessary.

Once a UTF-8 plain-text file is obtained, ParsCit searches for reference strings using a set of features. It first looks for section-heading tokens, which may include strings such as "References", "Bibliography", "References and Notes", or common variations, such as the reference-section string being split across multiple lines. If a token is found too early (by default before 40% of the document), the search continues. The finally selected location is taken as the start of the references section. Then it looks for tokens that mark the end, such as "appendices", "figures", "tables", "acknowledgments", "autobiographies", and so on, or the end of the document.

Once the references section has been identified, the next stage is to segment the reference strings. In general, references are marked in one of three ways: 1) numbers or strings enclosed in square brackets (e.g., “[1]”, “(1)”, “[Heckerman02]”); 2) plain numeric strings (e.g., "1", "1."); and 3) unmarked strings (e.g., in APA format). Therefore, the first step is to determine which marker style the citation list uses. For cases 1 and 2, a set of common marker patterns is constructed and matched with regular expressions; the number of matches for each pattern is then counted. If neither case matches at least one-sixth of the entries, the pattern with the highest match count is adopted. In both of these cases, the regular expressions used to detect the same marker type can also indicate the starting point of each reference entry and thereby split the reference list. If no citation markers are found, several heuristics are used to determine the start and end of each string. These include looking at the preceding line (a short line may be the last line of a reference string), lines that look like author-name lists (typically appearing before unmarked reference strings), and terminal punctuation (the final paragraph of a reference usually ends after a certain interval).

The segmented list is then fed independently into the CRF++ model described earlier and applied to the data.

4 Post-processing Steps

Based on the output produced by running CRF++, several post-processing steps are necessary. For example, each tagged field is converted to a standard representation. Author names can appear in various forms, such as “M.-Y. Kan and I. G. Councill” and “Kan, M.-Y. & Councill, I. G.” The name strings must first be split into components and positions (e.g., separated by commas or semicolons), and then normalized into forms such as “M-Y Kan” and “I G Councill”. Numeric fields, such as the volume and issue labels of a publication, are normalized so that only digits remain (e.g., "vol.5" is normalized to "5"). Likewise, only the year is retained from the date field. Finally, page numbers are normalized into a "start–end" format; for example, the field "pp. 584-589" becomes "584-589".

5 Extracting Citation Contexts

Based on the reference markers produced during post-processing, one or more regular expressions are generated and used to scan the body of the text. These expressions are based on the three marker types, corresponding to the three cases of reference-string segmentation described above. For explicit markers—strings enclosed in square brackets or parentheses—the marker is directly converted into a regular expression. For bare numeric markers (such as "1" or "1."), both bracketed and parenthesized expressions are generated, with square-bracket forms preferred; if a square-bracket form already exists, parenthesized expressions are not considered in the body text. The patterns are flexible enough to handle various situations in reference lists, such as grouped markers (e.g., "[12,2,5]") and mismatches with body content (e.g., "see Figure(2)").

Finally, for unmarked strings (such as in APA format), various marker forms are created from the authors' surnames and the publication year. Many marker formats may be generated. For example, Poljak, Rendl, and Wolkowicz (1994) might be represented as: 1) “Poljak, Rendl, Wolkowicz, 1994”, 2) “Poljak, Rendl, and Wolkowicz, 1994”, 3) “Poljak, Rendl, & Wolkowicz, 1994”, and 4) “Poljak et al., 1994”. Additional built-in expressions that increase flexibility by omitting punctuation will not be enumerated here. Each regular expression is applied to the body text, producing a list of all matching contexts. The size of the context string is configurable; by default, it extends 200 characters on either side of the match location. For long documents, the number of matches can be limited.

8 Related Work

The citation parsing problem has been addressed in earlier research (Cameron, 1997; Lawrence et al., 1999). Existing parsers we have examined generally fall into two categories: pattern matching and machine learning-based approaches.

Pattern matching methods require an input citation to be matched against known templates. The best matching template is used, and its tokens serve as citation field labels. A typical example of template-based methods is ParaTools (Jewell, 2000), a set of Perl modules that parse reference strings. ParaTools contains 400 templates for matching citation strings; however, even this large number is problematic. Although users can manually add new templates to ParaTools, the process is tedious and not scalable. In fact, authors may not strictly adhere to citation styles, or text extraction and OCR may produce strings that do not strictly match templates, which weakens the effectiveness of such programs. A further feature that reduces the utility of ParaTools is that it labels ambiguous fields as "any," effectively leaving them unannotated. Huang et al. (2004) report an accuracy of about 30% for ParaTools. This level of performance, combined with a lack of portability, makes such methods unsuitable for large-scale data processing.

The limitations of template-based methods have motivated researchers to explore machine learning models. Given sufficient training data, a machine learning-based parser can achieve high accuracy across different types of citations. We review four systems published in recent years that address this task.

Seymore et al. (1999) pioneered work on the Cora dataset. They used Hidden Markov Models (HMMs) to build a sequence labeler for bibliographic reference strings. Unlike standard HMMs, they used internal states to represent different regions, akin to the IOB encoding used in other tagging tasks. In later work, Peng and McCallum (2004) used the reference parsing task as a benchmark to test Conditional Random Fields (CRFs). Their work made CRFs the model of choice for this task. This motivated us to likewise choose CRF as the foundation for ParsCit's model.

The first version of ParsCit used Maximum Entropy (ME) training to build a model (Ng, 2004). Beyond using ME, which can be seen as a discriminative model one step beyond Hidden Markov Models, this work performed two-stage prediction: the first round labeled individual references, and the second round performed global labeling by considering how nearby reference strings (such as in the references or bibliography section) were labeled in the first round. This is the only approach that attempts to exploit such information—which may prove useful for maintaining a particular bibliographic style.

FLUX-CiM (Cortez et al., 2007) provides an unsupervised learning approach that uses word frequency adjustment to solve this problem. The method consists of four stages: truncation, alignment, binding, and joining. The final stage is analogous to ParsCit's last step, where extended fields (such as author) are segmented into components for that label. As mentioned earlier, FLUX-CiM performs noticeably better on journal and article titles than on "volume" and "author."

Citation context retrieval has long been an important feature of CiteSeer and nascent digital libraries. Heuristic-driven approaches have been continuously refined by these two systems and by Powley and Dale (2007). Using contextual cues around citations to extract summaries or to analyze how work is discussed by others continues to drive research in this area (Teufel et al., 2006; Wu et al., 2006; Schwartz et al., 2007).