1. class
  2. NoreSources
  3. \
  4. Text
  5. \
  6. Tokenifier

Tokenifier

class Tokenifier (View source)

Transforms a string to a list of tokens.

Properties

string $escape Escape character.
string[] $whitespaces White space character list.
string[] $escapables List of additional escapable characters.

Methods

addQuotingPair(stringown $start, string|NULL $end = null)

Add new quoting pair

setStringFunctions(callable $length, callable $split)

No description

clearQuotingPairs()

Remove all quoting pair definitions

string[]
__invoke(string $text)

No description

getEscapables($quotingPair)

No description

findStartingQuute($offset, $character)

No description

Details

at line 58
addQuotingPair(stringown $start, string|NULL $end = null)

Add new quoting pair

Parameters

stringown $start Opening quote
string|NULL $end Closing quoute. If NULL, use opening quote.

at line 73
setStringFunctions(callable $length, callable $split)

Parameters

callable $length strlen compatible function
callable $split str_split compatible function

at line 88
clearQuotingPairs()

Remove all quoting pair definitions

at line 99
string[] __invoke(string $text)

Parameters

string $text

Return Value

string[] List of tokens

Exceptions

InvalidArgumentException

at line 183
protected getEscapables($quotingPair)

Parameters

$quotingPair

at line 200
protected findStartingQuute($offset, $character)

Parameters

$offset
$character