Previously, only the string contents were stored in lexerState::token_str (i.e.
not including the delimeters). Now, the delimeters are stored as well, thus
preserving them. This also simplifies the code a bit.

A new function is added to handle the character storage, which is also now
applied for normal identifiers. To that end, the MAX_STRING_LENGTH was boosted
to 256 so that all reasonably sized identifiers may fit.