FINITE-STATE SOFTWARE TOOLS

Our authoring tools for finite-state linguistic components include:

• twolc Two-Level Rule Compiler

• lexc Finite-State Lexicon Compiler

• xfst Xerox Finite-State Tool.

These applications are used to create tokenizers, lexical transducer, noun phrase extractors, and other language-specific linguistic components. We share these tools with many other sites both inside and outside Xerox.

We have also created many runtime applications for tokenization, morphological analysis and disambiguation, including:

• tokenize Finite-State Tokenizer

• lookup Morphological Analyzer

• disamb HMM Disambiguator

All of these applications are built around a central library, called c-fsm. The library defines the data structures, provides the input/output routines, and implements the fundamental operations on finite-state networks. It is based on long-term Xerox research, originated by Ronald M. Kaplan and Martin Kay at PARC in the early 1980s.

The Xerox Linguistic Development Architecture (XeLDA) developed by SDE incorporates the CA finite-state technology.

Visit our Finite-State Home Page to learn about creating finite-state networks from regular expressions.

Back to Tools for Natural Language Processing