Treating escaped identifiers as character sequence tokens
Created by: ecoulson
The current lexer design parses escaped identifiers as character sequence tokens. The BNF definition of an escaped identifier is provided below.
escaped_identifier ::= \ {any_printable_ASCII_character_except_white_space} white_space
To solve this problem, a new token type called EscapedIdentifier should be introduced to the Token enum. This token will differentiate between character sequences and escaped identifiers. Otherwise, when we need to parse escaped identifiers, we would need to quickly check the string to see if it starts with a backslash (''). Adding this token type prevents that since we can simply create an error if it is not an EscapedIdentifier token. This may require splitting the lex_character_sequence method into two methods lex_character_sequence by introducing a new LexicalOperation for escaped identifiers.