CSLU Toolkit Formats


Grammar Format
Version: 1.0
Created:
25 May 2003
Modified:
25 May 2003

Overview
The Grammar format specifies grammars for speech recognition.    


Synopsis
FILE-BASED GRAMMAR
Tcl-LIST BASED GRAMMAR
token = rule ;
token = rule
lexicon <URI> ;
lexicon <URI>
// comment
--- comments are not allowed ---
/* comment */
--- comments are not allowed ---
clusterName := token1  token2  ...  tokenN ;
clusterName := token1  token2  ...  tokenN
token -> rule :: context __ context ;
token -> rule :: context __ context
--- blank lines ARE allowed --- --- blank lines are NOT allowed ---

Variables
Variable
Description
token
Any string of characters not including double-quotes.  If the token contains whitespace, then double quotes must surround the token.  Whitespace at the beginning and ending of the token is ignored.  The special characters described below may still be used in a token if they are preceded by a backslash.
rule
A sequence of token names and symbols that define how the token is to be expanded.  Valid symbols are ( and ) for grouping purposes; | to delimit the "or" operator between two tokens; [ and ] to indicate that whatever is within the brackets is optional; and < and > to identify a repeat operator.  Valid repeat operators are <+> (indicating that the previous item is to be repeated one or more times) and <*> (indicating that the previous item is to be repeated zero or more times).
URI
A uniform resource locator identifying the lexicon file.  Currently, only local URIs are supported.
comment
Any text.  A comment defined by // is terminated by the end of the line, and a comment defined by /* is terminated by */.  Comments are not allowed when the grammar is specified using a Tcl list.
clusterName
A cluster name identifes a group of tokens.  This group of tokens can then be represented by clusterName in a context description.
context
A context string is a set of one or more tokens (or cluster names) that identifies the left or right context of a rule.  The symbols __ (two adjacent underscore symbols) identify the center of the context.  If a context is "anything" (to the left or right of the center of the context), then the context string may be omitted.

Keywords and Reserved Symbols
Keyword or Symbol
Description
/BOU
When this keyword occurs in a context string or cluster, it identifies the beginning of the utterance.  It is important to identify /BOU as the left context of every state that may begin the grammar.  This may be done easily in most cases by putting the /BOU keyword in the same cluster as the silence model.
/EOU
When this keyword occurs in a context string or cluster, it identifies the end of the utterance.  It is important to identify /EOU as the right context of every state that may end the grammar.  This may be done easily in most cases by putting the /EOU keyword in the same cluster as the silence model.
/WB
When this keyword occurs in a context string, it specifies that the context must include a word boundary between the two tokens surrounding the /WB keyword.
/NWB
When this keyword occurs in a context string, it specifies that the context must NOT inclue a word boundary between the two tokens surrounding the /NWB keyword.
lexicon
The lexicon keyword at the beginning of a line, followed by a space, followed by a URI that is surrounded by angle brackets is reserved for defining a pronunciation dictionary of words.  If this keyword is encountered, the pronunciations for the words specified in the grammar will be obtained from the file indicated by the URI.
=
The equals sign defines the boundary between a token name and its rule.
;
A semicolon defines the end of a rule in the file-based form of a grammar.  In the Tcl-list based form of a grammar, the end of a rule is defined by the end of the list item.
< ... >
Angle brackets have two uses.  First, if a URI is expected (based on a preceding keyword), then the brackets surround the URI.  In a rule, brackets specify that the preceding item is to be repeated a number of times.  If the symbol inside the brackets is *, then the repetition will occur zero or more times.  If the symbol inside the brackets is +, then the repetition will occur one or more times.
//
When these characters occur in sequence in a grammar file, they define the beginning of a comment.  The comment ends when the end of the line is reached.  Anything within the comment is ignored by the Statenet package.  If the grammar is specified using Tcl lists, comments are not allowed.
/*  ... */
These characters define a comment in a grammar file.  The comment begins with /* and ends with */.  Anything within the comment is ignored by the Statenet package.  If the grammar is specified using Tcl lists, comments are not allowed.
:=
These symbols in sequence define the separation between a cluster name and a sequence of tokens that are to be clustered.
->
These symbols in sequence define the boundary between a token name and a context-dependent rule.
::
These symbols in sequence define the boundary between a context-dependent rule and the context in which the rule is applied.
__
These symbols in sequence define the boundary between the left and right context in a context-dependent rule.
( ... )
Rounded parentheses define a grouping of tokens within a rule.  Parentheses are useful if more than one word is to be repeated using the repeat operator <*> or <+>, or for grouping tokens with the "or" operator.
|
The "or" operator specifies that the token (or grouping) to the left of the | and the token (or grouping) to the right of the | have a parallel structure in the state network.  
[ ... ]
Square brackets identify whatever is within the brackets as optional.
\
The backslash may be used in front of any special character to prevent the parsing interpretations.  For example, the SAMPA symbol for the vowel in the word "bat" is {.  This symbol is reserved in ABNF form and will probably be reserved in this form as well in the future.  To ensure that this symbol will be treated as a token and not parsed, it may be preceded by a backslash, e.g. \{.
$
The dollar sign is NOT reserved.  However, it is considered good form to use token names that begin with a dollar sign at the level of the top grammar.


Description
The Grammar format specifies grammars for speech recognition.  This format is a modified version of the Augmented Backus-Naur Form (ABNF) developed for automatic speech recognition.  This modified format is read by the Statenet package and is used to create state networks for Hidden-Markov Model speech recognizers.  This format may be specified in a file or in a Tcl list; the format is slightly different in each case.  In particular, the Tcl list format does not allow comments, and the end of a rule, cluster assignment, or lexicon specification is specified by the end of the list item rather than by a semicolon.


Examples
The following example specifies continuous digit recognition at the level of the grammar, the lexicon (in Worldbet notation), and the context-dependent categories, in three separate files.  The context-dependent rules shown here are only a fraction of the entire set of rules for this recognizer.  The last box illustrates how the grammar files are read by the Statenet package.
 
digit.grammar:
$digit   = zero | oh | one | two | three | four | five | six |
           seven | eight | nine;
$grammar = [separator%%] ($digit [separator%%])<+> [separator%%];
digit.lexicon:
zero        = z I 9r oU              ;
oh          = oU                     ;
one         = w ^ n [&]              ;
two         = tc th u                ;
three       = T 9r i:                ;
four        = f oU 9r                ;
five        = f aI v                 ;
six         = s I kc kh s            ;
seven       = s E v I n [&]          ;
eight       = ei tc [th]             ;
nine        = n aI n [&]             ;

separator   = .pau [.garbage] .pau   ;

digit.phnrules:
$sil := .pau .garbage tc kc /BOU /EOU ;
$den_l := s z th ;
$den_r := s z th ;

// .pau -> <.pau> ;
.pau = <.pau> ;

// s -> $LC<s s>$RC :: $LC __ $RC ;

s -> v<s s>E :: v __ E ;

s -> n<s s>E :: n __ E ;

s -> &<s s>E :: & __ E ;

s -> 9r<s s>E :: 9r __ E ;
grammarExample1.tcl
package require Statenet

set stateNet [statenet create digit.grammar "grammar"]
statenet add $stateNet digit.lexicon "word"
statenet add $stateNet digit.phnrules "phoneme" -selfLoops 1

statenet print $stateNet

nuke $stateNet



The next example illustrates a context-dependent rule that expands t into tc (t-closure) followed by an optional th (t-burst), or the flapped phoneme d_\( (note the use of backslash to prevent parsing the parenthesis in the phoneme name), when the t occurs in the context of b and ^ on the left and 3r on the right.  This causes the pronunciation of word1 to be /b ^ (tc [th]) | d_\( 3r/ or /b I t 3r/ or /b E t 3r/, and the pronunciation of word2 is /m ^ t 3r/, where only the t in the specified context has been expanded.

cdexample1.txt:
butter = b (^ | I | E) t 3r;
mutter = m ^ t 3r;

$grammar = butter | mutter;
cdexample2.txt:
t        -> (tc [th]) | d_\(    :: b ^ __ 3r ;
grammarExample2.tcl:
package require Statenet

set stateNet [statenet create cdexample1.txt "grammar"]
statenet add $stateNet cdexample2.txt "word"
statenet print $stateNet

nuke $stateNet



See Also
The Statenet package

Author
John-Paul Hosom, hosom@{cslu, bme, cse}.ogi.edu