DocTokenizer¶
class DocTokenizer
Documentation comment tokenizer.
Variables¶
| Name | Description |
|---|---|
| iter | Iterator over source. |
| end | End of source. |
Functions¶
| Name | Description |
|---|---|
| DocTokenizer | Constructor. |
| next | Get the next token. |
| consume | Consume tokens until stopping criterion. |
Variable Details¶
iter¶
std::string_view::const_iterator iter
Iterator over source.
end¶
std::string_view::const_iterator end
End of source.
Function Details¶
DocTokenizer¶
DocTokenizer(const std::string_view& source)
Constructor.
-
comment - Comment to tokenize.
next¶
DocToken next()
Get the next token.
- Return
- Next token.
The token is only valid for the lifetime of the DocTokenizer, as it
contains a reference to a substring of the source file. If no tokens
remain (i.e. the end of the file is reached), a token with a type of
NONE is returned.
consume¶
DocToken consume(const int stop = ANY)
Consume tokens until stopping criterion.
-
stop - Bitmask giving the token types on which to stop and return.
- Return
- Last token consumed.
Tokens are consumed until one is encountered with a type in stop,
which is then returned. If stop is ANY then the next token is
returned.