Package: rtiktoken 0.0.6

David Zimmermann-Kollenda

rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

Authors:David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut], Authors of the dependent Rust crates [aut]

rtiktoken_0.0.6.tar.gz
rtiktoken_0.0.6.zip(r-4.5)rtiktoken_0.0.6.zip(r-4.4)rtiktoken_0.0.6.zip(r-4.3)
rtiktoken_0.0.6.tgz(r-4.5-x86_64)rtiktoken_0.0.6.tgz(r-4.5-arm64)rtiktoken_0.0.6.tgz(r-4.4-x86_64)rtiktoken_0.0.6.tgz(r-4.4-arm64)rtiktoken_0.0.6.tgz(r-4.3-x86_64)rtiktoken_0.0.6.tgz(r-4.3-arm64)
rtiktoken_0.0.6.tar.gz(r-4.5-noble)rtiktoken_0.0.6.tar.gz(r-4.4-noble)
rtiktoken.pdf |rtiktoken.html
rtiktoken/json (API)
NEWS

# Install 'rtiktoken' in R:
install.packages('rtiktoken', repos = c('https://davzim.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/davzim/rtiktoken/issues

Pkgdown site:https://davzim.github.io

On CRAN:

Conda:

rustcargo

4.22 score 11 stars 3 scripts 547 downloads 4 exports 0 dependencies

Last updated 4 months agofrom:3cb11358a3. Checks:11 OK. Indexed: yes.

TargetResultLatest binary
Doc / VignettesOKFeb 22 2025
R-4.5-win-x86_64OKFeb 22 2025
R-4.5-mac-x86_64OKFeb 22 2025
R-4.5-mac-aarch64OKFeb 22 2025
R-4.5-linux-x86_64OKFeb 22 2025
R-4.4-win-x86_64OKFeb 22 2025
R-4.4-mac-x86_64OKFeb 22 2025
R-4.4-mac-aarch64OKFeb 22 2025
R-4.3-win-x86_64OKFeb 22 2025
R-4.3-mac-x86_64OKFeb 22 2025
R-4.3-mac-aarch64OKFeb 22 2025

Exports:decode_tokensget_token_countget_tokensmodel_to_tokenizer

Dependencies: