Package: rtiktoken 0.11.0-1

David Zimmermann-Kollenda

rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

Authors:David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut], Authors of the dependent Rust crates [aut]

rtiktoken_0.11.0-1.tar.gz
rtiktoken_0.11.0-1.zip(r-4.7)rtiktoken_0.11.0-1.zip(r-4.6)rtiktoken_0.11.0-1.zip(r-4.5)
rtiktoken_0.11.0-1.tgz(r-4.6-x86_64)rtiktoken_0.11.0-1.tgz(r-4.6-arm64)rtiktoken_0.11.0-1.tgz(r-4.5-x86_64)rtiktoken_0.11.0-1.tgz(r-4.5-arm64)
rtiktoken_0.11.0-1.tar.gz(r-4.7-arm64)rtiktoken_0.11.0-1.tar.gz(r-4.7-x86_64)rtiktoken_0.11.0-1.tar.gz(r-4.6-arm64)rtiktoken_0.11.0-1.tar.gz(r-4.6-x86_64)
rtiktoken_0.11.0.tgz(r-4.5-emscripten)
manual.pdf |manual.html✨
card.svg |card.png
rtiktoken/json (API)
NEWS

# Install 'rtiktoken' in R:
install.packages('rtiktoken', repos = c('https://davzim.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/davzim/rtiktoken/issues

Pkgdown/docs site:https://davzim.github.io

On CRAN:

Conda:

bpeopenairusttokenizationcargo

4.15 score 14 stars 7 scripts 156 downloads 4 exports 0 dependencies

Last updated from:52e7f16a92. Checks:12 OK, 1 FAIL. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-arm64OK160
linux-devel-x86_64OK136
source / vignettesOK199
linux-release-arm64OK149
linux-release-x86_64OK149
macos-release-arm64OK123
macos-release-x86_64OK309
macos-oldrel-arm64OK116
macos-oldrel-x86_64OK246
windows-develOK177
windows-releaseOK169
windows-oldrelOK174
wasm-releaseFAIL153

Exports:decode_tokensget_token_countget_tokensmodel_to_tokenizer

Dependencies: