Interface IWordState

Defines an interface for tokenizer state that processes words, identificators or keywords

Hierarchy

ITokenizerState
- IWordState

Implemented by

Index

Methods

Methods

clearWordChars

clearWordChars(): void

- Defined in tokenizers/IWordState.ts:22
Clears definitions of word chars.

Returns void

nextToken

nextToken(scanner: IScanner, tokenizer: ITokenizer): Token

Inherited from ITokenizerState.nextToken
- Defined in tokenizers/ITokenizerState.ts:23
Gets the next token from the stream started from the character linked to this state.

Parameters
- scanner: IScanner
  
  A textual string to be tokenized.
- tokenizer: ITokenizer
  
  A tokenizer class that controls the process.
Returns Token

The next token from the top of the stream.

setWordChars

setWordChars(fromSymbol: number, toSymbol: number, enable: boolean): void

- Defined in tokenizers/IWordState.ts:17
Establish characters in the given range as valid characters for part of a word after the first character. Note that the tokenizer must determine which characters are valid as the beginning character of a word.

Parameters
- fromSymbol: number
  
  First character index of the interval.
- toSymbol: number
  
  Last character index of the interval.
- enable: boolean
  
  true if this state should use characters in the given range.
Returns void

Generated using TypeDoc