Class CsvTokenizer

Implements a tokenizer class for CSV files.

Hierarchy

AbstractTokenizer
- CsvTokenizer

Implements

ITokenizer

Index

Constructors

constructor

Properties

Accessors

Methods

Constructors

constructor

new CsvTokenizer(): CsvTokenizer

Overrides AbstractTokenizer.constructor
- Defined in csv/CsvTokenizer.ts:114
Constructs this object with default parameters.

Returns CsvTokenizer

Properties

Protected _lastTokenType

_lastTokenType: TokenType = TokenType.Unknown

Protected _nextToken

_nextToken: Token

Protected _scanner

_scanner: IScanner

commentState

commentState: ICommentState

decodeStrings

decodeStrings: boolean

mergeWhitespaces

mergeWhitespaces: boolean

numberState

numberState: INumberState

quoteState

quoteState: IQuoteState

skipComments

skipComments: boolean

skipEof

skipEof: boolean

skipUnknown

skipUnknown: boolean

skipWhitespaces

skipWhitespaces: boolean

symbolState

symbolState: ISymbolState

unifyNumbers

unifyNumbers: boolean

whitespaceState

whitespaceState: IWhitespaceState

wordState

wordState: IWordState

Accessors

endOfLine

get endOfLine(): string
set endOfLine(value: string): void

- Defined in csv/CsvTokenizer.ts:54
Separator for rows in CSV stream.

Returns string
- Defined in csv/CsvTokenizer.ts:61
Separator for rows in CSV stream.

Parameters
- value: string
Returns void

fieldSeparators

get fieldSeparators(): number[]
set fieldSeparators(value: number[]): void

- Defined in csv/CsvTokenizer.ts:20
Separator for fields in CSV stream.

Returns number[]
- Defined in csv/CsvTokenizer.ts:27
Separator for fields in CSV stream.

Parameters
- value: number[]
Returns void

quoteSymbols

get quoteSymbols(): number[]
set quoteSymbols(value: number[]): void

- Defined in csv/CsvTokenizer.ts:68
Character to quote strings.

Returns number[]
- Defined in csv/CsvTokenizer.ts:75
Character to quote strings.

Parameters
- value: number[]
Returns void

scanner

get scanner(): IScanner
set scanner(value: IScanner): void

Inherited from AbstractTokenizer.scanner
- Defined in tokenizers/AbstractTokenizer.ts:58
Returns IScanner
Inherited from AbstractTokenizer.scanner
- Defined in tokenizers/AbstractTokenizer.ts:62
Parameters
- value: IScanner
Returns void

Methods

clearCharacterStates

clearCharacterStates(): void

Inherited from AbstractTokenizer.clearCharacterStates
- Defined in tokenizers/AbstractTokenizer.ts:54
Returns void

getCharacterState

getCharacterState(symbol: number): ITokenizerState

Inherited from AbstractTokenizer.getCharacterState
- Defined in tokenizers/AbstractTokenizer.ts:46
Parameters
- symbol: number
Returns ITokenizerState

hasNextToken

hasNextToken(): boolean

Implementation of ITokenizer.hasNextToken

Inherited from AbstractTokenizer.hasNextToken
- Defined in tokenizers/AbstractTokenizer.ts:68
Returns boolean

nextToken

nextToken(): Token

Implementation of ITokenizer.nextToken

Inherited from AbstractTokenizer.nextToken
- Defined in tokenizers/AbstractTokenizer.ts:73
Returns Token

Protected readNextToken

readNextToken(): Token

Inherited from AbstractTokenizer.readNextToken
- Defined in tokenizers/AbstractTokenizer.ts:79
Returns Token

setCharacterState

setCharacterState(fromSymbol: number, toSymbol: number, state: ITokenizerState): void

Inherited from AbstractTokenizer.setCharacterState
- Defined in tokenizers/AbstractTokenizer.ts:50
Parameters
- fromSymbol: number
- toSymbol: number
- state: ITokenizerState
Returns void

tokenizeBuffer

tokenizeBuffer(buffer: string): Token[]

Implementation of ITokenizer.tokenizeBuffer

Inherited from AbstractTokenizer.tokenizeBuffer
- Defined in tokenizers/AbstractTokenizer.ts:168
Parameters
- buffer: string
Returns Token[]

tokenizeBufferToStrings

tokenizeBufferToStrings(buffer: string): string[]

Implementation of ITokenizer.tokenizeBufferToStrings

Inherited from AbstractTokenizer.tokenizeBufferToStrings
- Defined in tokenizers/AbstractTokenizer.ts:182
Parameters
- buffer: string
Returns string[]

tokenizeStream

tokenizeStream(scanner: IScanner): Token[]

Implementation of ITokenizer.tokenizeStream

Inherited from AbstractTokenizer.tokenizeStream
- Defined in tokenizers/AbstractTokenizer.ts:159
Parameters
- scanner: IScanner
Returns Token[]

tokenizeStreamToStrings

tokenizeStreamToStrings(scanner: IScanner): string[]

Implementation of ITokenizer.tokenizeStreamToStrings

Inherited from AbstractTokenizer.tokenizeStreamToStrings
- Defined in tokenizers/AbstractTokenizer.ts:173
Parameters
- scanner: IScanner
Returns string[]

Generated using TypeDoc