Class: Lexer

Lexer

new Lexer()

Converts source code into a token stream.
Source:

Methods

getErrors() → {Array}

Returns errors found during lexing
Source:
Returns:
Errors
Type
Array

lex(source) → {Array}

Traverses the source code input to verify and build a token stream.
Parameters:
Name Type Description
source String The source code to convert into tokens
Source:
Returns:
An array of tokens
Type
Array

<inner> findToken(str) → {String}

Finds a token at the beginning of the source input.
Parameters:
Name Type Description
str String A chunk of source code to be tested for a token at the start.
Source:
Returns:
A token constant that represents the type of token found.
Type
String

<inner> getTokenLength(tokenType) → {Integer}

Gets the length of a token.
Parameters:
Name Type Description
tokenType String Type of token to find the length of
Source:
Returns:
length of token or 1 if token is not found
Type
Integer

<inner> process(src) → {Array}

Each recursive itteration will have one less token removed from the front of the source.
Parameters:
Name Type Description
src String Source code to process.
Source:
Returns:
Stream of tokens found in source.
Type
Array