decode_tokens {rtiktoken}R Documentation

Decodes tokens back to text

Description

Decodes tokens back to text

Usage

decode_tokens(tokens, model)

Arguments

tokens

a vector of tokens to decode, or a list of tokens

model

a model to use for tokenization, either a model name, e.g., ⁠gpt-4o⁠ or a tokenizer, e.g., o200k_base. See also available tokenizers.

Value

a character string of the decoded tokens or a vector or strings

See Also

model_to_tokenizer(), get_tokens()

Examples

tokens <- get_tokens("Hello World", "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")

tokens <- get_tokens(c("Hello World", "Alice Bob Charlie"), "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")

[Package rtiktoken version 0.0.6 Index]