René's URL Explorer Experiment


Title: GitHub - agentstation/tokenizer: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling.

Open Graph Title: GitHub - agentstation/tokenizer: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling.

X Title: GitHub - agentstation/tokenizer: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling.

Description: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling. - agentstation/tokenizer

Open Graph Description: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling. - agentstation/tokenizer

X Description: High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling. - agentstation/tokenizer

Opengraph URL: https://github.com/agentstation/tokenizer

X: @github

direct link

Domain: patch-diff.githubusercontent.com

route-pattern/:user_id/:repository
route-controllerfiles
route-actiondisambiguate
fetch-noncev2:523657c2-2bac-5182-973c-4f0b7f48ed3b
current-catalog-service-hashf3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-idAA42:7B515:63AC1F:87C238:69809DA3
html-safe-nonce9b458fa7318cd34952bd8f6609247961e3ff753217602cabff18607f4f6f68ae
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQTQyOjdCNTE1OjYzQUMxRjo4N0MyMzg6Njk4MDlEQTMiLCJ2aXNpdG9yX2lkIjoiMjk0MDA3NDEwOTk2MDIzMjM1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac32f646ac38c584921f06f67ae7abdf840e31653b32aa5979deaba5b807ba7df9
hovercard-subject-tagrepository:1029484812
github-keyboard-shortcutsrepository,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location//
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/agentstation/tokenizer
twitter:imagehttps://opengraph.githubassets.com/b522839ef3052499e4e79e401ea6d7bfeba025b54d919258b4191e0d2c18df1b/agentstation/tokenizer
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/b522839ef3052499e4e79e401ea6d7bfeba025b54d919258b4191e0d2c18df1b/agentstation/tokenizer
og:image:altHigh-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling. - agentstation/tokenizer
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Nonefc372bc1bc3274494294cdd18b3fe5c3e061c47ebf976706ceab8e6b341155bc
turbo-cache-controlno-preview
go-importgithub.com/agentstation/tokenizer git https://github.com/agentstation/tokenizer.git
octolytics-dimension-user_id165715156
octolytics-dimension-user_loginagentstation
octolytics-dimension-repository_id1029484812
octolytics-dimension-repository_nwoagentstation/tokenizer
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id1029484812
octolytics-dimension-repository_network_root_nwoagentstation/tokenizer
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasecaf470b206a280846518c5dee1e18f46cefc391b
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://patch-diff.githubusercontent.com/agentstation/tokenizer#start-of-content
https://patch-diff.githubusercontent.com/
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2Fagentstation%2Ftokenizer
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2Fagentstation%2Ftokenizer
Sign up https://patch-diff.githubusercontent.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=agentstation%2Ftokenizer
Reloadhttps://patch-diff.githubusercontent.com/agentstation/tokenizer
Reloadhttps://patch-diff.githubusercontent.com/agentstation/tokenizer
Reloadhttps://patch-diff.githubusercontent.com/agentstation/tokenizer
agentstation https://patch-diff.githubusercontent.com/agentstation
tokenizerhttps://patch-diff.githubusercontent.com/agentstation/tokenizer
Notifications https://patch-diff.githubusercontent.com/login?return_to=%2Fagentstation%2Ftokenizer
Fork 0 https://patch-diff.githubusercontent.com/login?return_to=%2Fagentstation%2Ftokenizer
Star 0 https://patch-diff.githubusercontent.com/login?return_to=%2Fagentstation%2Ftokenizer
pkg.go.dev/github.com/agentstation/tokenizerhttps://pkg.go.dev/github.com/agentstation/tokenizer
MIT license https://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/LICENSE
0 stars https://patch-diff.githubusercontent.com/agentstation/tokenizer/stargazers
0 forks https://patch-diff.githubusercontent.com/agentstation/tokenizer/forks
Branches https://patch-diff.githubusercontent.com/agentstation/tokenizer/branches
Tags https://patch-diff.githubusercontent.com/agentstation/tokenizer/tags
Activity https://patch-diff.githubusercontent.com/agentstation/tokenizer/activity
Star https://patch-diff.githubusercontent.com/login?return_to=%2Fagentstation%2Ftokenizer
Notifications https://patch-diff.githubusercontent.com/login?return_to=%2Fagentstation%2Ftokenizer
Code https://patch-diff.githubusercontent.com/agentstation/tokenizer
Issues 0 https://patch-diff.githubusercontent.com/agentstation/tokenizer/issues
Pull requests 9 https://patch-diff.githubusercontent.com/agentstation/tokenizer/pulls
Actions https://patch-diff.githubusercontent.com/agentstation/tokenizer/actions
Projects 0 https://patch-diff.githubusercontent.com/agentstation/tokenizer/projects
Security 0 https://patch-diff.githubusercontent.com/agentstation/tokenizer/security
Insights https://patch-diff.githubusercontent.com/agentstation/tokenizer/pulse
Code https://patch-diff.githubusercontent.com/agentstation/tokenizer
Issues https://patch-diff.githubusercontent.com/agentstation/tokenizer/issues
Pull requests https://patch-diff.githubusercontent.com/agentstation/tokenizer/pulls
Actions https://patch-diff.githubusercontent.com/agentstation/tokenizer/actions
Projects https://patch-diff.githubusercontent.com/agentstation/tokenizer/projects
Security https://patch-diff.githubusercontent.com/agentstation/tokenizer/security
Insights https://patch-diff.githubusercontent.com/agentstation/tokenizer/pulse
Brancheshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/branches
Tagshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tags
https://patch-diff.githubusercontent.com/agentstation/tokenizer/branches
https://patch-diff.githubusercontent.com/agentstation/tokenizer/tags
65 Commitshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/commits/master/
https://patch-diff.githubusercontent.com/agentstation/tokenizer/commits/master/
.githubhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/.github
.githubhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/.github
cmdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/cmd
cmdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/cmd
docshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/docs
docshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/docs
llama3https://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/llama3
llama3https://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/llama3
scriptshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/scripts
scriptshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/tree/master/scripts
.air.tomlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.air.toml
.air.tomlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.air.toml
.env.examplehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.env.example
.env.examplehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.env.example
.gitignorehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.gitignore
.gitignorehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.gitignore
.godot.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.godot.yaml
.godot.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.godot.yaml
.golangci.ymlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.golangci.yml
.golangci.ymlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.golangci.yml
.goreleaser.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.goreleaser.yaml
.goreleaser.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.goreleaser.yaml
.pre-commit-config.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.pre-commit-config.yaml
.pre-commit-config.yamlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/.pre-commit-config.yaml
CLAUDE.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/CLAUDE.md
CLAUDE.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/CLAUDE.md
Dockerfilehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/Dockerfile
Dockerfilehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/Dockerfile
LICENSEhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/LICENSE
LICENSEhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/LICENSE
Makefilehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/Makefile
Makefilehttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/Makefile
README.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/README.md
README.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/README.md
RELEASING.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/RELEASING.md
RELEASING.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/RELEASING.md
codecov.ymlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/codecov.yml
codecov.ymlhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/codecov.yml
devbox.jsonhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/devbox.json
devbox.jsonhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/devbox.json
devbox.lockhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/devbox.lock
devbox.lockhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/devbox.lock
generate.gohttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/generate.go
generate.gohttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/generate.go
go.modhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/go.mod
go.modhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/go.mod
go.sumhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/go.sum
go.sumhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/go.sum
READMEhttps://patch-diff.githubusercontent.com/agentstation/tokenizer
MIT licensehttps://patch-diff.githubusercontent.com/agentstation/tokenizer
https://patch-diff.githubusercontent.com/agentstation/tokenizer#tokenizer
https://pkg.go.dev/github.com/agentstation/tokenizer
https://goreportcard.com/report/github.com/agentstation/tokenizer
https://github.com/agentstation/tokenizer/actions
https://codecov.io/gh/agentstation/tokenizer
https://raw.githubusercontent.com/agentstation/tokenizer/master/LICENSE
https://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/docs/demo.svg
https://patch-diff.githubusercontent.com/agentstation/tokenizer#installation
https://patch-diff.githubusercontent.com/agentstation/tokenizer#using-homebrew-recommended-for-macoslinux
https://patch-diff.githubusercontent.com/agentstation/tokenizer#download-binary
releases pagehttps://github.com/agentstation/tokenizer/releases/latest
https://patch-diff.githubusercontent.com/agentstation/tokenizer#using-go
https://patch-diff.githubusercontent.com/agentstation/tokenizer#build-from-source
https://patch-diff.githubusercontent.com/agentstation/tokenizer#cli-tool
cmd/tokenizer/README.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/cmd/tokenizer/README.md
https://patch-diff.githubusercontent.com/agentstation/tokenizer#library-packages
https://patch-diff.githubusercontent.com/agentstation/tokenizer#llama3
llama3/README.mdhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/blob/master/llama3/README.md
https://patch-diff.githubusercontent.com/agentstation/tokenizer#installation-1
https://patch-diff.githubusercontent.com/agentstation/tokenizer#quick-start
https://patch-diff.githubusercontent.com/agentstation/tokenizer#cli-usage
https://patch-diff.githubusercontent.com/agentstation/tokenizer#library-usage
https://patch-diff.githubusercontent.com/agentstation/tokenizer#development
https://patch-diff.githubusercontent.com/agentstation/tokenizer#prerequisites
Devboxhttps://www.jetify.com/devbox
https://patch-diff.githubusercontent.com/agentstation/tokenizer#setup-development-environment
https://patch-diff.githubusercontent.com/agentstation/tokenizer#common-development-tasks
https://patch-diff.githubusercontent.com/agentstation/tokenizer#release-process
https://patch-diff.githubusercontent.com/agentstation/tokenizer#contributing
https://patch-diff.githubusercontent.com/agentstation/tokenizer#license
https://patch-diff.githubusercontent.com/agentstation/tokenizer#tokenizer-1
https://patch-diff.githubusercontent.com/agentstation/tokenizer#index
gomarkdochttps://github.com/princjef/gomarkdoc
pkg.go.dev/github.com/agentstation/tokenizerhttps://pkg.go.dev/github.com/agentstation/tokenizer
nlp https://patch-diff.githubusercontent.com/topics/nlp
cli https://patch-diff.githubusercontent.com/topics/cli
golang https://patch-diff.githubusercontent.com/topics/golang
machine-learning https://patch-diff.githubusercontent.com/topics/machine-learning
natural-language-processing https://patch-diff.githubusercontent.com/topics/natural-language-processing
ai https://patch-diff.githubusercontent.com/topics/ai
tokenizer https://patch-diff.githubusercontent.com/topics/tokenizer
text-processing https://patch-diff.githubusercontent.com/topics/text-processing
goreleaser https://patch-diff.githubusercontent.com/topics/goreleaser
bpe https://patch-diff.githubusercontent.com/topics/bpe
llm https://patch-diff.githubusercontent.com/topics/llm
llama3 https://patch-diff.githubusercontent.com/topics/llama3
gomarkdoc https://patch-diff.githubusercontent.com/topics/gomarkdoc
Readme https://patch-diff.githubusercontent.com/agentstation/tokenizer#readme-ov-file
MIT license https://patch-diff.githubusercontent.com/agentstation/tokenizer#MIT-1-ov-file
Please reload this pagehttps://patch-diff.githubusercontent.com/agentstation/tokenizer
Activityhttps://patch-diff.githubusercontent.com/agentstation/tokenizer/activity
Custom propertieshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/custom-properties
0 starshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/stargazers
0 watchinghttps://patch-diff.githubusercontent.com/agentstation/tokenizer/watchers
0 forkshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/forks
Report repository https://patch-diff.githubusercontent.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fagentstation%2Ftokenizer&report=agentstation+%28user%29
Releases 8https://patch-diff.githubusercontent.com/agentstation/tokenizer/releases
v0.0.8 Latest Sep 6, 2025 https://patch-diff.githubusercontent.com/agentstation/tokenizer/releases/tag/v0.0.8
+ 7 releaseshttps://patch-diff.githubusercontent.com/agentstation/tokenizer/releases
Packages 0https://patch-diff.githubusercontent.com/orgs/agentstation/packages?repo_name=tokenizer
Contributors 3https://patch-diff.githubusercontent.com/agentstation/tokenizer/graphs/contributors
Please reload this pagehttps://patch-diff.githubusercontent.com/agentstation/tokenizer
Go 94.7% https://patch-diff.githubusercontent.com/agentstation/tokenizer/search?l=go
Makefile 3.1% https://patch-diff.githubusercontent.com/agentstation/tokenizer/search?l=makefile
Shell 2.1% https://patch-diff.githubusercontent.com/agentstation/tokenizer/search?l=shell
Dockerfile 0.1% https://patch-diff.githubusercontent.com/agentstation/tokenizer/search?l=dockerfile
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.