get_token_count {rtiktoken} | R Documentation |
Returns the number of tokens in a text
Description
Returns the number of tokens in a text
Usage
get_token_count(text, model)
Arguments
text |
a character string to encode to tokens, can be a vector |
model |
a model to use for tokenization, either a model name, e.g., |
Value
the number of tokens in the text, vector of integers
See Also
model_to_tokenizer()
, get_tokens()
Examples
get_token_count("Hello World", "gpt-4o")
[Package rtiktoken version 0.0.6 Index]