model_to_tokenizer {rtiktoken} | R Documentation |
Gets the name of the tokenizer used by a model
Description
Gets the name of the tokenizer used by a model
Usage
model_to_tokenizer(model)
Arguments
model |
the model to use, e.g., |
Value
the tokenizer used by the model
Examples
model_to_tokenizer("gpt-4o")
model_to_tokenizer("gpt-4-1106-preview")
model_to_tokenizer("text-davinci-002")
model_to_tokenizer("text-embedding-ada-002")
model_to_tokenizer("text-embedding-3-small")
[Package rtiktoken version 0.0.6 Index]