René's URL Explorer Experiment


Title: GitHub - lenML/tokenizers: a lightweight no-dependency fork from transformers.js (only tokenizers)

Open Graph Title: GitHub - lenML/tokenizers: a lightweight no-dependency fork from transformers.js (only tokenizers)

X Title: GitHub - lenML/tokenizers: a lightweight no-dependency fork from transformers.js (only tokenizers)

Description: a lightweight no-dependency fork from transformers.js (only tokenizers) - lenML/tokenizers

Open Graph Description: a lightweight no-dependency fork from transformers.js (only tokenizers) - lenML/tokenizers

X Description: a lightweight no-dependency fork from transformers.js (only tokenizers) - lenML/tokenizers

Opengraph URL: https://github.com/lenML/tokenizers

X: @github

direct link

Domain: patch-diff.githubusercontent.com

route-pattern/:user_id/:repository
route-controllerfiles
route-actiondisambiguate
fetch-noncev2:12d2993c-6023-eb95-cce6-846d975162bd
current-catalog-service-hashf3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id8A6C:7AF86:111BE68:1674F0D:698D6201
html-safe-noncec87ecec02bd182cdc50ce0dfac88d25a4a7c9c5d9e4d0cb694ee63985777b2da
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4QTZDOjdBRjg2OjExMUJFNjg6MTY3NEYwRDo2OThENjIwMSIsInZpc2l0b3JfaWQiOiI2MjIxNzc0NDg2NzIwMzA3NzEzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacf0187d142c52e448857d0de5168d04912268736a6ccf8b58b92b1684f11d5380
hovercard-subject-tagrepository:789341239
github-keyboard-shortcutsrepository,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location//
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/lenML/tokenizers
twitter:imagehttps://opengraph.githubassets.com/584e7dbdbaa18ca4b2c2b01c937ef94d945dfdf3b1879897edf6f12465287583/lenML/tokenizers
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/584e7dbdbaa18ca4b2c2b01c937ef94d945dfdf3b1879897edf6f12465287583/lenML/tokenizers
og:image:alta lightweight no-dependency fork from transformers.js (only tokenizers) - lenML/tokenizers
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Nonec0818105fa276287e9369cfdefa0a0fa7953719791ceff9b94d69623c0a4fe8a
turbo-cache-controlno-preview
go-importgithub.com/lenML/tokenizers git https://github.com/lenML/tokenizers.git
octolytics-dimension-user_id154655365
octolytics-dimension-user_loginlenML
octolytics-dimension-repository_id789341239
octolytics-dimension-repository_nwolenML/tokenizers
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id789341239
octolytics-dimension-repository_network_root_nwolenML/tokenizers
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea95a17cc440c14d4fcddc0641bc1136fa8d908f0
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://patch-diff.githubusercontent.com/lenML/tokenizers#start-of-content
https://patch-diff.githubusercontent.com/
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2FlenML%2Ftokenizers
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2FlenML%2Ftokenizers
Sign up https://patch-diff.githubusercontent.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=lenML%2Ftokenizers
Reloadhttps://patch-diff.githubusercontent.com/lenML/tokenizers
Reloadhttps://patch-diff.githubusercontent.com/lenML/tokenizers
Reloadhttps://patch-diff.githubusercontent.com/lenML/tokenizers
lenML https://patch-diff.githubusercontent.com/lenML
tokenizershttps://patch-diff.githubusercontent.com/lenML/tokenizers
Notifications https://patch-diff.githubusercontent.com/login?return_to=%2FlenML%2Ftokenizers
Fork 1 https://patch-diff.githubusercontent.com/login?return_to=%2FlenML%2Ftokenizers
Star 32 https://patch-diff.githubusercontent.com/login?return_to=%2FlenML%2Ftokenizers
MIT license https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/LICENSE
32 stars https://patch-diff.githubusercontent.com/lenML/tokenizers/stargazers
1 fork https://patch-diff.githubusercontent.com/lenML/tokenizers/forks
Branches https://patch-diff.githubusercontent.com/lenML/tokenizers/branches
Tags https://patch-diff.githubusercontent.com/lenML/tokenizers/tags
Activity https://patch-diff.githubusercontent.com/lenML/tokenizers/activity
Star https://patch-diff.githubusercontent.com/login?return_to=%2FlenML%2Ftokenizers
Notifications https://patch-diff.githubusercontent.com/login?return_to=%2FlenML%2Ftokenizers
Code https://patch-diff.githubusercontent.com/lenML/tokenizers
Issues 1 https://patch-diff.githubusercontent.com/lenML/tokenizers/issues
Pull requests 0 https://patch-diff.githubusercontent.com/lenML/tokenizers/pulls
Actions https://patch-diff.githubusercontent.com/lenML/tokenizers/actions
Security 0 https://patch-diff.githubusercontent.com/lenML/tokenizers/security
Insights https://patch-diff.githubusercontent.com/lenML/tokenizers/pulse
Code https://patch-diff.githubusercontent.com/lenML/tokenizers
Issues https://patch-diff.githubusercontent.com/lenML/tokenizers/issues
Pull requests https://patch-diff.githubusercontent.com/lenML/tokenizers/pulls
Actions https://patch-diff.githubusercontent.com/lenML/tokenizers/actions
Security https://patch-diff.githubusercontent.com/lenML/tokenizers/security
Insights https://patch-diff.githubusercontent.com/lenML/tokenizers/pulse
Brancheshttps://patch-diff.githubusercontent.com/lenML/tokenizers/branches
Tagshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tags
https://patch-diff.githubusercontent.com/lenML/tokenizers/branches
https://patch-diff.githubusercontent.com/lenML/tokenizers/tags
70 Commitshttps://patch-diff.githubusercontent.com/lenML/tokenizers/commits/main/
https://patch-diff.githubusercontent.com/lenML/tokenizers/commits/main/
.github/ISSUE_TEMPLATEhttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/.github/ISSUE_TEMPLATE
.github/ISSUE_TEMPLATEhttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/.github/ISSUE_TEMPLATE
.vscodehttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/.vscode
.vscodehttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/.vscode
assetshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/assets
assetshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/assets
exampleshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/examples
exampleshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/examples
packageshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/packages
packageshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/packages
pretrainedhttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/pretrained
pretrainedhttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/pretrained
scriptshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/scripts
scriptshttps://patch-diff.githubusercontent.com/lenML/tokenizers/tree/main/scripts
.gitignorehttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/.gitignore
.gitignorehttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/.gitignore
LICENSEhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/LICENSE
LICENSEhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/LICENSE
README.mdhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/README.md
README.mdhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/README.md
package.jsonhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/package.json
package.jsonhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/package.json
pnpm-lock.yamlhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/pnpm-lock.yaml
pnpm-lock.yamlhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/pnpm-lock.yaml
pnpm-workspace.yamlhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/pnpm-workspace.yaml
pnpm-workspace.yamlhttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/pnpm-workspace.yaml
READMEhttps://patch-diff.githubusercontent.com/lenML/tokenizers
MIT licensehttps://patch-diff.githubusercontent.com/lenML/tokenizers
https://patch-diff.githubusercontent.com/lenML/tokenizers#lenmltokenizers
huggingface/transformers.jshttps://github.com/huggingface/transformers.js/
https://patch-diff.githubusercontent.com/lenML/tokenizers#tokenizer-arena--playground
click to arena pagehttps://lenml.github.io/tokenizers-arena/#/side-by-side
https://lenml.github.io/tokenizers-arena/#/side-by-side
https://patch-diff.githubusercontent.com/lenML/tokenizers#when-should-i-use-this-instead-of-transformersjs
https://patch-diff.githubusercontent.com/lenML/tokenizers#packages
@lenml/tokenizershttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/tokenizers
https://www.npmjs.com/package/@lenml/tokenizers
https://camo.githubusercontent.com/35ea4e47490ad4f06a0cdca355f8c6c097237e5de749c960775203e862e8be8c/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a657273
@lenml/tokenizer-qwen3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/qwen3
https://www.npmjs.com/package/@lenml/tokenizer-qwen3
https://camo.githubusercontent.com/8f8e8f1fe49d45e6d231d15f6cefff64fe50c11da765889ee3352eb78e5076d8/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d7177656e33
@lenml/tokenizer-gptosshttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gptoss
https://www.npmjs.com/package/@lenml/tokenizer-gptoss
https://camo.githubusercontent.com/2c53c114ee8ee30dadfc06688da9e1b52b9539abbe7288d99df366ac38fb1dc8/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6770746f7373
@lenml/tokenizer-minicpm_v4_5https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/minicpm_v4_5
https://www.npmjs.com/package/@lenml/tokenizer-minicpm_v4_5
https://camo.githubusercontent.com/9d761c2bd99f1c9728246f5b13e28ef2344403724d56b76977ce954b878e5e2c/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6d696e6963706d5f76345f35
@lenml/tokenizer-gemma3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gemma3
https://www.npmjs.com/package/@lenml/tokenizer-gemma3
https://camo.githubusercontent.com/8b6f0675b5ebf64cea368714d209ca02a68beeedff0c0cdccb3681828ccc61a6/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67656d6d6133
@lenml/tokenizer-deepseek_v3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/deepseek_v3
https://www.npmjs.com/package/@lenml/tokenizer-deepseek_v3
https://camo.githubusercontent.com/fa3af7d4edfa0048f1cc2947e7fc72c0e1b2336d7d8a9364aea7b9db8b850c38/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d646565707365656b5f7633
@lenml/tokenizer-llama3_1https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/llama3_1
https://www.npmjs.com/package/@lenml/tokenizer-llama3_1
https://camo.githubusercontent.com/c850ca23ba0752d88d12014ad9c6dc5dd6cd94a213dcc4470bdeae0ec0d68697/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6c6c616d61335f31
@lenml/tokenizer-llama2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/llama2
https://www.npmjs.com/package/@lenml/tokenizer-llama2
https://camo.githubusercontent.com/c56bc115ab503999221f2607e2d7215ae8235ec588ec9d4ac00437f8d59ec7bd/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6c6c616d6132
@lenml/tokenizer-llama3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/llama3
https://www.npmjs.com/package/@lenml/tokenizer-llama3
https://camo.githubusercontent.com/2f8a99fbbf40ba538692ec24958b8f6c6c97b2a1c6a8b2ba114c645eb3f318db/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6c6c616d6133
@lenml/tokenizer-gpt4ohttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt4o
https://www.npmjs.com/package/@lenml/tokenizer-gpt4o
https://camo.githubusercontent.com/d51a7685fdeb4031e99176827c2ad4465913c60105cf1c74dc1b82aac224828f/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d677074346f
@lenml/tokenizer-gpt4https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt4
https://www.npmjs.com/package/@lenml/tokenizer-gpt4
https://camo.githubusercontent.com/a5a11e92fafb72df76ef8d143545f9cdf3526b65b081fcd0a7027aba1778b4c4/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67707434
@lenml/tokenizer-gpt35turbohttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt35turbo
https://www.npmjs.com/package/@lenml/tokenizer-gpt35turbo
https://camo.githubusercontent.com/e317d83f43c8b3fc6f78da751f6f56cfe59bbd3d969157f6fd5c565210f40a9d/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6770743335747572626f
@lenml/tokenizer-gpt35turbo16khttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt35turbo16k
https://www.npmjs.com/package/@lenml/tokenizer-gpt35turbo16k
https://camo.githubusercontent.com/a8f9451749487ea3cfbc89e293dde33b87b2915a6372551a4d86c9d3f9d0847c/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6770743335747572626f31366b
@lenml/tokenizer-gpt3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt3
https://www.npmjs.com/package/@lenml/tokenizer-gpt3
https://camo.githubusercontent.com/fb355f53b2f8c5a31c454c0bfd82cd9320a326483b287204952b975eb4d20be9/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67707433
@lenml/tokenizer-gemmahttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gemma
https://www.npmjs.com/package/@lenml/tokenizer-gemma
https://camo.githubusercontent.com/6f55b226d54d22b2e020f538a81459dcc64611f4a2ecc7318a9cb734bcd881f2/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67656d6d61
@lenml/tokenizer-claudehttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/claude
https://www.npmjs.com/package/@lenml/tokenizer-claude
https://camo.githubusercontent.com/4b3a0e0f7c6b5c35a141c3da4bae3122dc909a272836a235b28c089af615ad9f/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d636c61756465
@lenml/tokenizer-claude1https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/claude1
https://www.npmjs.com/package/@lenml/tokenizer-claude1
https://camo.githubusercontent.com/f1049c088793cd3e4699f6db7dc5e79df121c9fb622cfc0043ac357cb156ddc3/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d636c6175646531
@lenml/tokenizer-gpt2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gpt2
https://www.npmjs.com/package/@lenml/tokenizer-gpt2
https://camo.githubusercontent.com/bf6dd5f5fd49ec32d2c857d1052ccf5544005082eebb03bfcdb27fe8911daf12/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67707432
@lenml/tokenizer-baichuan2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/baichuan2
https://www.npmjs.com/package/@lenml/tokenizer-baichuan2
https://camo.githubusercontent.com/a29fbe55fb255c62c47e8c486b84c8d45e747b9e34dd28c71e0d90356904708f/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d626169636875616e32
@lenml/tokenizer-chatglm3https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/chatglm3
https://www.npmjs.com/package/@lenml/tokenizer-chatglm3
https://camo.githubusercontent.com/1edbd593d28065b546d3aea39b958a542427a0e796be1e8a6be1aa93a46289eb/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d63686174676c6d33
@lenml/tokenizer-command_r_plushttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/command_r_plus
https://www.npmjs.com/package/@lenml/tokenizer-command_r_plus
https://camo.githubusercontent.com/7bbd1e65bf7d09249db7364f358021de4f01333fc1ba1b3f000aaaa8d1222085/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d636f6d6d616e645f725f706c7573
@lenml/tokenizer-internlm2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/internlm2
https://www.npmjs.com/package/@lenml/tokenizer-internlm2
https://camo.githubusercontent.com/32b7a94d1591b1ada419a527ede271b72e83f74febdf2a9b33cd5caaf9f845b7/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d696e7465726e6c6d32
@lenml/tokenizer-qwen1_5https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/qwen1_5
https://www.npmjs.com/package/@lenml/tokenizer-qwen1_5
https://camo.githubusercontent.com/6fe7c4c6812abe19807725dbbb3d00baaa89471594d80365d83981fdaa0079b7/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d7177656e315f35
@lenml/tokenizer-yihttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/yi
https://www.npmjs.com/package/@lenml/tokenizer-yi
https://camo.githubusercontent.com/bcbf519dffbef234e035f7913e5d979886f4b9f79d26dfb888662bfb61402f95/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d7969
@lenml/tokenizer-text_davinci002https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/text_davinci002
https://www.npmjs.com/package/@lenml/tokenizer-text_davinci002
https://camo.githubusercontent.com/60708b3729441e2f3c930027ff38656be4d3c9811f6f8c8c3ea642230557c041/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d746578745f646176696e6369303032
@lenml/tokenizer-text_davinci003https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/text_davinci003
https://www.npmjs.com/package/@lenml/tokenizer-text_davinci003
https://camo.githubusercontent.com/c85d4f6ecc0e73318220049a01b2971df6fb967ac10e6b456b5778a6280b664e/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d746578745f646176696e6369303033
@lenml/tokenizer-text_embedding_ada002https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/text_embedding_ada002
https://www.npmjs.com/package/@lenml/tokenizer-text_embedding_ada002
https://camo.githubusercontent.com/5292fd3c66f9b7d620b3f1e0fc9a4d7da61be3d056a3f6722df99a6453cf966b/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d746578745f656d62656464696e675f616461303032
@lenml/tokenizer-gemma2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gemma2
https://www.npmjs.com/package/@lenml/tokenizer-gemma2
https://camo.githubusercontent.com/4eba35487085a3b340fad8f5f4a9b3f2b7e464147d5014d3921be18b936b81c9/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67656d6d6132
@lenml/tokenizer-geminihttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/gemini
https://www.npmjs.com/package/@lenml/tokenizer-gemini
https://camo.githubusercontent.com/53a2c22bffb363949d907bcd030a735b05f997314dc901ee3c2833f4f7c1af5c/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d67656d696e69
@lenml/tokenizer-qwen2_5https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/qwen2_5
https://www.npmjs.com/package/@lenml/tokenizer-qwen2_5
https://camo.githubusercontent.com/401ccb753b175b1e258b9e4c056154c952d58effbadc508158943bc2a9ab4866/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d7177656e325f35
@lenml/tokenizer-aya_expansehttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/aya_expanse
https://www.npmjs.com/package/@lenml/tokenizer-aya_expanse
https://camo.githubusercontent.com/4fea727f5a31454a1bdb19d1cb7e5fc6e49a028d7a4c1c3f9d22325c35a1bcad/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6179615f657870616e7365
@lenml/tokenizer-llama3_2https://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/llama3_2
https://www.npmjs.com/package/@lenml/tokenizer-llama3_2
https://camo.githubusercontent.com/267379dd3c282e29415ed003d7824a2ce53643c62591f42939a726273ec7127a/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6c6c616d61335f32
@lenml/tokenizer-mistral_nemohttps://patch-diff.githubusercontent.com/lenML/tokenizers/blob/main/packages/mistral_nemo
https://www.npmjs.com/package/@lenml/tokenizer-mistral_nemo
https://camo.githubusercontent.com/e926fff164ff6866f73f359ec2de9dd9d83de922206ecacd97014165f28bf12b/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f64772f2534306c656e6d6c253246746f6b656e697a65722d6d69737472616c5f6e656d6f
https://patch-diff.githubusercontent.com/lenML/tokenizers#usage
https://patch-diff.githubusercontent.com/lenML/tokenizers#install
https://patch-diff.githubusercontent.com/lenML/tokenizers#npmyarnpnpm
https://patch-diff.githubusercontent.com/lenML/tokenizers#esm
https://patch-diff.githubusercontent.com/lenML/tokenizers#load-tokenizer
https://patch-diff.githubusercontent.com/lenML/tokenizers#from-json
https://patch-diff.githubusercontent.com/lenML/tokenizers#from-urls
https://patch-diff.githubusercontent.com/lenML/tokenizers#from-pre-packaged-tokenizer
https://patch-diff.githubusercontent.com/lenML/tokenizers#chat-template
https://patch-diff.githubusercontent.com/lenML/tokenizers#tokenizer-api
transformer.js tokenizers documenthttps://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers
https://patch-diff.githubusercontent.com/lenML/tokenizers#get-lightweight-transformerstokenizers
https://patch-diff.githubusercontent.com/lenML/tokenizers#manual-packaging
https://patch-diff.githubusercontent.com/lenML/tokenizers#performance-benchmark-results
https://patch-diff.githubusercontent.com/lenML/tokenizers#related-repo
https://github.com/huggingface/transformers.jshttps://github.com/huggingface/transformers.js
https://github.com/huggingface/tokenizers.jshttps://github.com/huggingface/tokenizers.js
https://patch-diff.githubusercontent.com/lenML/tokenizers#license
tokenizer https://patch-diff.githubusercontent.com/topics/tokenizer
mistral https://patch-diff.githubusercontent.com/topics/mistral
baichuan https://patch-diff.githubusercontent.com/topics/baichuan
gpt4 https://patch-diff.githubusercontent.com/topics/gpt4
transfomers https://patch-diff.githubusercontent.com/topics/transfomers
chatgpt https://patch-diff.githubusercontent.com/topics/chatgpt
chatglm https://patch-diff.githubusercontent.com/topics/chatglm
llama2 https://patch-diff.githubusercontent.com/topics/llama2
llama3 https://patch-diff.githubusercontent.com/topics/llama3
Readme https://patch-diff.githubusercontent.com/lenML/tokenizers#readme-ov-file
MIT license https://patch-diff.githubusercontent.com/lenML/tokenizers#MIT-1-ov-file
Please reload this pagehttps://patch-diff.githubusercontent.com/lenML/tokenizers
Activityhttps://patch-diff.githubusercontent.com/lenML/tokenizers/activity
Custom propertieshttps://patch-diff.githubusercontent.com/lenML/tokenizers/custom-properties
32 starshttps://patch-diff.githubusercontent.com/lenML/tokenizers/stargazers
1 watchinghttps://patch-diff.githubusercontent.com/lenML/tokenizers/watchers
1 forkhttps://patch-diff.githubusercontent.com/lenML/tokenizers/forks
Report repository https://patch-diff.githubusercontent.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FlenML%2Ftokenizers&report=lenML+%28user%29
Please reload this pagehttps://patch-diff.githubusercontent.com/lenML/tokenizers
JavaScript 93.1% https://patch-diff.githubusercontent.com/lenML/tokenizers/search?l=javascript
TypeScript 6.7% https://patch-diff.githubusercontent.com/lenML/tokenizers/search?l=typescript
Python 0.2% https://patch-diff.githubusercontent.com/lenML/tokenizers/search?l=python
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.