Class AbstractTokenizer

Implements an abstract tokenizer class.

Hierarchy

AbstractTokenizer

Implements

ITokenizer

Index

Constructors

constructor

Properties

Accessors

scanner

Methods

Constructors

Protected constructor

new AbstractTokenizer(): AbstractTokenizer

- Defined in tokenizers/AbstractTokenizer.ts:41
Returns AbstractTokenizer

Properties

Protected _lastTokenType

_lastTokenType: TokenType = TokenType.Unknown

Protected _nextToken

_nextToken: Token

Protected _scanner

_scanner: IScanner

commentState

commentState: ICommentState

decodeStrings

decodeStrings: boolean

mergeWhitespaces

mergeWhitespaces: boolean

numberState

numberState: INumberState

quoteState

quoteState: IQuoteState

skipComments

skipComments: boolean

skipEof

skipEof: boolean

skipUnknown

skipUnknown: boolean

skipWhitespaces

skipWhitespaces: boolean

symbolState

symbolState: ISymbolState

unifyNumbers

unifyNumbers: boolean

whitespaceState

whitespaceState: IWhitespaceState

wordState

wordState: IWordState

Accessors

scanner

get scanner(): IScanner
set scanner(value: IScanner): void

- Defined in tokenizers/AbstractTokenizer.ts:58
Returns IScanner
- Defined in tokenizers/AbstractTokenizer.ts:62
Parameters
- value: IScanner
Returns void

Methods

clearCharacterStates

clearCharacterStates(): void

- Defined in tokenizers/AbstractTokenizer.ts:54
Returns void

getCharacterState

getCharacterState(symbol: number): ITokenizerState

- Defined in tokenizers/AbstractTokenizer.ts:46
Parameters
- symbol: number
Returns ITokenizerState

hasNextToken

hasNextToken(): boolean

Implementation of ITokenizer.hasNextToken
- Defined in tokenizers/AbstractTokenizer.ts:68
Returns boolean

nextToken

nextToken(): Token

Implementation of ITokenizer.nextToken
- Defined in tokenizers/AbstractTokenizer.ts:73
Returns Token

Protected readNextToken

readNextToken(): Token

- Defined in tokenizers/AbstractTokenizer.ts:79
Returns Token

setCharacterState

setCharacterState(fromSymbol: number, toSymbol: number, state: ITokenizerState): void

- Defined in tokenizers/AbstractTokenizer.ts:50
Parameters
- fromSymbol: number
- toSymbol: number
- state: ITokenizerState
Returns void

tokenizeBuffer

tokenizeBuffer(buffer: string): Token[]

Implementation of ITokenizer.tokenizeBuffer
- Defined in tokenizers/AbstractTokenizer.ts:168
Parameters
- buffer: string
Returns Token[]

tokenizeBufferToStrings

tokenizeBufferToStrings(buffer: string): string[]

Implementation of ITokenizer.tokenizeBufferToStrings
- Defined in tokenizers/AbstractTokenizer.ts:182
Parameters
- buffer: string
Returns string[]

tokenizeStream

tokenizeStream(scanner: IScanner): Token[]

Implementation of ITokenizer.tokenizeStream
- Defined in tokenizers/AbstractTokenizer.ts:159
Parameters
- scanner: IScanner
Returns Token[]

tokenizeStreamToStrings

tokenizeStreamToStrings(scanner: IScanner): string[]

Implementation of ITokenizer.tokenizeStreamToStrings
- Defined in tokenizers/AbstractTokenizer.ts:173
Parameters
- scanner: IScanner
Returns string[]

Generated using TypeDoc