Tokenizer¶
class Tokenizer
Documentation comment tokenizer.
Variables¶
| Name | Description |
|---|---|
| iter | Iterator over source. |
| end | End of source. |
Functions¶
| Name | Description |
|---|---|
| Tokenizer | Constructor. |
| next | Get the next token. |
| consume | Consume tokens until stopping criterion. |
Variable Details¶
end¶
std::string_view::const_iterator end
End of source.
iter¶
std::string_view::const_iterator iter
Iterator over source.
Function Details¶
Tokenizer¶
Tokenizer(const std::string_view& source)
Constructor.
-
comment - Comment to tokenize.
consume¶
Token consume(const int stop = ANY)
Consume tokens until stopping criterion.
-
stop - Bitmask giving the token type on which to stop and return.
- Return
- Last token consumed.
Tokens are consumed until one is encountered with a type in stop,
which is then returned. If stop is ANY then the next token is
returned.
- See
- interpret()
next¶
Token next()
Get the next token.
- Return
- Next token.
The token is only valid for the lifetime of the Tokenizer, as it contains
a reference to a substring of the source file. If no tokens remain (i.e.
the end of the file is reached), a token with a type of NONE is
returned.