Treating escaped identifiers as character sequence tokens
Created by: ecoulson
The current lexer design parses escaped identifiers as character sequence tokens. The BNF definition of an escaped identifier is provided below.
escaped_identifier ::= \ {any_printable_ASCII_character_except_white_space} white_space
To solve this problem, a new token type called EscapedIdentifier
should be introduced to the Token
enum. This token will differentiate between character sequences and escaped identifiers. Otherwise, when we need to parse escaped identifiers, we would need to quickly check the string to see if it starts with a backslash (''). Adding this token type prevents that since we can simply create an error if it is not an EscapedIdentifier
token. This may require splitting the lex_character_sequence
method into two methods lex_character_sequence
by introducing a new LexicalOperation
for escaped identifiers.