decode_tokens {rtiktoken} | R Documentation |
Decodes tokens back to text
Description
Decodes tokens back to text
Usage
decode_tokens(tokens, model)
Arguments
tokens |
a vector of tokens to decode, or a list of tokens |
model |
a model to use for tokenization, either a model name, e.g., |
Value
a character string of the decoded tokens or a vector or strings
See Also
model_to_tokenizer()
, get_tokens()
Examples
tokens <- get_tokens("Hello World", "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")
tokens <- get_tokens(c("Hello World", "Alice Bob Charlie"), "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")
[Package rtiktoken version 0.0.6 Index]