From method names, I am compelled to believe you are creating some sort of a Lexer object. Generally you implement Lexer with stratified design. First consumption of characters, then tokens (made out of characters), then optionally constructs made out of tokens.
Hope this helps.
