Lexer
Related Docs:
object Lexer
| package string
Lazily separate a string into tokens based on some boundary condition and preserve the delimiter characters that were between the boundaries, including any leading or trailing delimiters.
Lazily separate a string into tokens based on some boundary condition and preserve the delimiter characters that were between the boundaries, including any leading or trailing delimiters.
string to separate
a lazy result
Separate a string into tokens based on some boundary condition ignoring any delimiter characters between any two tokens.
Separate a string into tokens based on some boundary condition ignoring any delimiter characters between any two tokens.
string to separate into tokens
Iterator of tokens
A trait for a lexer that allows separating a string into "tokens" and "delimiters". Each lexer defines tokens and delimiters differently. A whitespace lexer assigns whitespace runs as delimiters and non-whitespace runs as tokens. A capitalization-based lexer creates tokens when capitalization changes from lower case to upper case but produces no delimiters.