Options
All
  • Public
  • Public/Protected
  • All
Menu

Class GenericWordState

A wordState returns a word from a scanner. Like other states, a tokenizer transfers the job of reading to this state, depending on an initial character. Thus, the tokenizer decides which characters may begin a word, and this state determines which characters may appear as a second or later character in a word. These are typically different sets of characters; in particular, it is typical for digits to appear as parts of a word, but not as the initial character of a word.

By default, the following characters may appear in a word. The method setWordChars() allows customizing this.

From    To
  'a', 'z'
  'A', 'Z'
  '0', '9'

as well as: minus sign, underscore, and apostrophe.

Hierarchy

Implements

Index

Constructors

Methods

Constructors

constructor

  • Constructs a word state with a default idea of what characters are admissible inside a word (as described in the class comment).

    Returns GenericWordState

Methods

clearWordChars

  • clearWordChars(): void

  • Clears definitions of word chars.

    Returns void

nextToken

  • Ignore word (such as blanks and tabs), and return the tokenizer's next token.

    Parameters

    • scanner: IScanner

      A textual string to be tokenized.

    • tokenizer: ITokenizer

      A tokenizer class that controls the process.

    Returns Token

    The next token from the top of the stream.

setWordChars

  • setWordChars(fromSymbol: number, toSymbol: number, enable: boolean): void

  • Establish characters in the given range as valid characters for part of a word after the first character. Note that the tokenizer must determine which characters are valid as the beginning character of a word.

    Parameters

    • fromSymbol: number

      First character index of the interval.

    • toSymbol: number

      Last character index of the interval.

    • enable: boolean

      true if this state should use characters in the given range.

    Returns void

Generated using TypeDoc