Skip to content

Include whitespace and comment tokens

thebirdsdontsing requested to merge fix/include-whitespace-and-comment-tokens into main

Created by: ecoulson

Summary

This PR introduces whitespace and comments tokens to the token stream during lexing. This is necessary because identifiers and numbers are not necessarily composed of one token. Because of this, there may be white space and comments between tokens that compose identifiers and numbers. Thus, the parser must be able to emit parsing errors when a whitespace or comment exists in an invalid position.

Merge request reports