Add tests for lexer
Created by: ecoulson
What?
Added tests for lexing empty files, white space, comments, numbers, string literals, character sequences, operators, keywords, and punctuation.
Why?
As per #8 (closed), confidence that the tokens produced by the lexer are correct matters since parsing tokens incorrect tokens would hinder progress in developing the parser.
How?
Implemented unit tests for the lexer that cover a majority of the tokens that can be output by the tokenization process
Testing?
Added coverage for all lexing functions.
Anything Else?
See #10 (closed) for a change that needs to be implemented due to writing these tests.