Package: rtiktoken 0.11.0.3

David Zimmermann-Kollenda

rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

Authors:David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut], Authors of the dependent Rust crates [aut]

rtiktoken_0.11.0.3.tar.gz
rtiktoken_0.11.0.3.zip(r-4.7-x86_64)rtiktoken_0.11.0.3.zip(r-4.7-arm64)rtiktoken_0.11.0.3.zip(r-4.6-x86_64)rtiktoken_0.11.0.3.zip(r-4.6-arm64)rtiktoken_0.11.0.3.zip(r-4.5-x86_64)
rtiktoken_0.11.0.3.tgz(r-4.6-x86_64)rtiktoken_0.11.0.3.tgz(r-4.6-arm64)rtiktoken_0.11.0.3.tgz(r-4.5-x86_64)rtiktoken_0.11.0.3.tgz(r-4.5-arm64)
rtiktoken_0.11.0.3.tar.gz(r-4.7-arm64)rtiktoken_0.11.0.3.tar.gz(r-4.7-x86_64)rtiktoken_0.11.0.3.tar.gz(r-4.6-arm64)rtiktoken_0.11.0.3.tar.gz(r-4.6-x86_64)
rtiktoken_0.11.0.3.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
rtiktoken/json (API)

# Install 'rtiktoken' in R:
install.packages('rtiktoken', repos = c('https://davzim.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/davzim/rtiktoken/issues

Pkgdown/docs site:https://davzim.github.io

On CRAN:

Conda:

bpeopenairusttokenizationcargo

4.45 score 14 stars 7 scripts 331 downloads 4 exports 0 dependencies

Last updated from:73ffc2d6f3. Checks:15 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-arm64OK164
linux-devel-x86_64OK141
source / vignettesOK251
linux-release-arm64OK142
linux-release-x86_64OK156
macos-release-arm64OK269
macos-release-x86_64OK335
macos-oldrel-arm64OK205
macos-oldrel-x86_64OK341
windows-devel-arm64OK399
windows-devel-x86_64OK190
windows-release-arm64OK408
windows-release-x86_64OK163
windows-oldrel-x86_64OK150
wasm-releaseOK142

Exports:decode_tokensget_token_countget_tokensmodel_to_tokenizer

Dependencies: