Token¶
class Token
Token.
A token is only valid for the lifetime of the Tokenizer that produced it, as it contains a reference to a substring of the source file.
Variables¶
| Name | Description |
|---|---|
| type | Token type. |
| first | Iterator to first character. |
| last | Iterator to one-past-last character. |
Functions¶
| Name | Description |
|---|---|
| Token | Constructor. |
| str | Get token as string. |
| substr | Get substring of the token as a string. |
Variable Details¶
first¶
std::string_view::const_iterator first
Iterator to first character.
last¶
std::string_view::const_iterator last
Iterator to one-past-last character.
type¶
TokenType type
Token type.
Function Details¶
Token¶
Token(const TokenType& type, const std::string_view::const_iterator& first, const std::string_view::const_iterator& last)
Constructor.
-
type - Token type.
-
first - Iterator to first character in source.
-
last - Iterator to one-past-last character in source.
str¶
std::string_view str() const
Get token as string.
substr¶
std::string_view substr(size_t pos = 0) const
Get substring of the token as a string.
-
pos - Position of the first character.