get_tokens {rtiktoken} | R Documentation |
Converts text to tokens
Description
Converts text to tokens
Usage
get_tokens(text, model)
Arguments
text |
a character string to encode to tokens, can be a vector |
model |
a model to use for tokenization, either a model name, e.g., |
Value
a vector of tokens for the given text as integer
See Also
model_to_tokenizer()
, decode_tokens()
Examples
get_tokens("Hello World", "gpt-4o")
get_tokens("Hello World", "o200k_base")
[Package rtiktoken version 0.0.6 Index]