model_to_tokenizer {rtiktoken}R Documentation

Gets the name of the tokenizer used by a model

Description

Gets the name of the tokenizer used by a model

Usage

model_to_tokenizer(model)

Arguments

model

the model to use, e.g., ⁠gpt-4o⁠

Value

the tokenizer used by the model

Examples

model_to_tokenizer("gpt-4o")
model_to_tokenizer("gpt-4-1106-preview")
model_to_tokenizer("text-davinci-002")
model_to_tokenizer("text-embedding-ada-002")
model_to_tokenizer("text-embedding-3-small")

[Package rtiktoken version 0.0.6 Index]