Package: rtiktoken 0.0.6
David Zimmermann-Kollenda
rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Authors:
rtiktoken_0.0.6.tar.gz
rtiktoken_0.0.6.zip(r-4.5)rtiktoken_0.0.6.zip(r-4.4)rtiktoken_0.0.6.zip(r-4.3)
rtiktoken_0.0.6.tgz(r-4.4-x86_64)rtiktoken_0.0.6.tgz(r-4.4-arm64)rtiktoken_0.0.6.tgz(r-4.3-x86_64)rtiktoken_0.0.6.tgz(r-4.3-arm64)
rtiktoken_0.0.6.tar.gz(r-4.5-noble)rtiktoken_0.0.6.tar.gz(r-4.4-noble)
rtiktoken.pdf |rtiktoken.html✨
rtiktoken/json (API)
NEWS
| # Install 'rtiktoken' in R: |
| install.packages('rtiktoken', repos = c('https://davzim.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/davzim/rtiktoken/issues
Last updated 12 days agofrom:3cb11358a3. Checks:OK: 9. Indexed: yes.
| Target | Result | Date |
|---|---|---|
| Doc / Vignettes | OK | Nov 07 2024 |
| R-4.5-win-x86_64 | OK | Nov 07 2024 |
| R-4.5-linux-x86_64 | OK | Nov 07 2024 |
| R-4.4-win-x86_64 | OK | Nov 07 2024 |
| R-4.4-mac-x86_64 | OK | Nov 07 2024 |
| R-4.4-mac-aarch64 | OK | Nov 07 2024 |
| R-4.3-win-x86_64 | OK | Nov 07 2024 |
| R-4.3-mac-x86_64 | OK | Nov 07 2024 |
| R-4.3-mac-aarch64 | OK | Nov 07 2024 |
Exports:decode_tokensget_token_countget_tokensmodel_to_tokenizer
Dependencies:
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| Decodes tokens back to text | decode_tokens |
| Returns the number of tokens in a text | get_token_count |
| Converts text to tokens | get_tokens |
| Gets the name of the tokenizer used by a model | model_to_tokenizer |