René's URL Explorer Experiment

Title: GitHub - bbulb/trawl: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server) · GitHub

Open Graph Title: GitHub - bbulb/trawl: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server)

X Title: GitHub - bbulb/trawl: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server)

Description: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server) - bbulb/trawl

Open Graph Description: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server) - bbulb/trawl

X Description: Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server) - bbulb/trawl

Opengraph URL: https://github.com/bbulb/trawl

X: @github

direct link

Domain: github.com

react-profiling	0
react-import-map	react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:fb396a4b-99e9-db4a-73d8-41078c464c96
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	BFFE:CBF7F:8F83AD:BC6502:6AEBED25
html-safe-nonce	109b3643a69a87c9c2ed7522875d59812a42206dc61377d6f165a0421f520cb9
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRkZFOkNCRjdGOjhGODNBRDpCQzY1MDI6NkFFQkVEMjUiLCJ2aXNpdG9yX2lkIjoiNDM1NjM3Nzk0Mjc4Mjg5NzQ0NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac	3266e98501d7f9cb90b9621deb3ea19a538b19a9de4db196bfc9227401caaabc
hovercard-subject-tag	repository:1210008589
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/bbulb/trawl
twitter:image	https://opengraph.githubassets.com/59374802d025d704280365c3a2c9f03e714d7e11bd597b87dec5ae51ca299e72/bbulb/trawl
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/59374802d025d704280365c3a2c9f03e714d7e11bd597b87dec5ae51ca299e72/bbulb/trawl
og:image:alt	Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server) - bbulb/trawl
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	8ec2990e713e4b47a35fdedb81d463ccadec4d3a4ba7253c2513727b0a58ecf9
turbo-cache-control	no-cache
go-import	github.com/bbulb/trawl git https://github.com/bbulb/trawl.git
octolytics-dimension-user_id	76148831
octolytics-dimension-user_login	bbulb
octolytics-dimension-repository_id	1210008589
octolytics-dimension-repository_nwo	bbulb/trawl
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	false
octolytics-dimension-repository_network_root_id	1210008589
octolytics-dimension-repository_network_root_nwo	bbulb/trawl
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	ddbc1c308788e31a0779acbd2fdbf652c948e2ef
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/bbulb/trawl#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fbbulb%2Ftrawl
GitHubRepository hosting, code review, and collaboration	https://github.com/platform
CopilotAI coding agent for tasks, projects and automated work	https://github.com/features/copilot
ActionsCI/CD and automation	https://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcement	https://github.com/features/code-quality
CodespacesCloud development environments	https://github.com/features/codespaces
Code SecurityVulnerability detection and remediation	https://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak prevention	https://github.com/security/advanced-security/secret-protection
All Features	https://github.com/features
Documentation	https://docs.github.com/en
Training	https://learn.github.com/skills
Certifications	https://learn.github.com/credentials
Build with Copilot	https://github.com/features/ai
Customize Copilot	https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrations	https://github.com/marketplace
Trending repos	https://github.com/trending
GitHub Sponsors	https://github.com/open-source/sponsors
Developer articles	https://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.	https://github.blog/changelog/
Enterprises	https://github.com/enterprise
Team	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
See all industries	https://github.com/solutions/industry
Why GitHub	https://github.com/why-github
Customer Stories	https://github.com/customer-stories
Business insights	https://github.com/solutions/executive-insights
Trust Center	https://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code security	https://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/enterprise/premium-support
Documentation	https://docs.github.com/en
Resource center	https://github.com/resources
Blog	https://github.blog/
Events & webinars	https://github.com/resources/events
Community Forum	https://github.com/orgs/community/discussions
Customer Stories	https://github.com/customer-stories
Integrations	https://github.com/marketplace
Customer support	https://help.github.com/
Training	https://learn.github.com/skills
GitHub Status	https://www.githubstatus.com/
About	https://github.com/about
Partners	https://github.com/partners
Changelog	https://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.	https://githubuniverse.com/
Pricing	https://github.com/pricing
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fbbulb%2Ftrawl
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=bbulb%2Ftrawl
Reload	https://github.com/bbulb/trawl
Reload	https://github.com/bbulb/trawl
Reload	https://github.com/bbulb/trawl
bbulb	https://github.com/bbulb
trawl	https://github.com/bbulb/trawl
Code	https://github.com/bbulb/trawl
Issues	https://github.com/bbulb/trawl/issues
Pull requests	https://github.com/bbulb/trawl/pulls
Actions	https://github.com/bbulb/trawl/actions
Projects	https://github.com/bbulb/trawl/projects
Security and quality	https://github.com/bbulb/trawl/security
Insights	https://github.com/bbulb/trawl/pulse
	https://github.com/bbulb/trawl
trawl	https://github.com/bbulb/trawl
Notifications	https://github.com/login?return_to=%2Fbbulb%2Ftrawl
Fork0 (0)	https://github.com/login?return_to=%2Fbbulb%2Ftrawl
Star3 (3)	https://github.com/login?return_to=%2Fbbulb%2Ftrawl
ai-agents	https://github.com/topics/ai-agents
bge-m3	https://github.com/topics/bge-m3
content-extraction	https://github.com/topics/content-extraction
embeddings	https://github.com/topics/embeddings
llm	https://github.com/topics/llm
mcp	https://github.com/topics/mcp
mcp-server	https://github.com/topics/mcp-server
playwright	https://github.com/topics/playwright
python	https://github.com/topics/python
rag	https://github.com/topics/rag
trafilatura	https://github.com/topics/trafilatura
web-scraping	https://github.com/topics/web-scraping
Readme	https://github.com/bbulb/trawl#readme-ov-file
MIT license	https://github.com/bbulb/trawl#MIT-1-ov-file
Contributing	https://github.com/bbulb/trawl#contributing-ov-file
Activity	https://github.com/bbulb/trawl/activity
Custom properties	https://github.com/bbulb/trawl/custom-properties
0 forks	https://github.com/bbulb/trawl/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fbbulb%2Ftrawl&report=bbulb+%28user%29
Branches	https://github.com/bbulb/trawl/branches
Tags	https://github.com/bbulb/trawl/tags
	https://github.com/bbulb/trawl/branches
	https://github.com/bbulb/trawl/tags
214 Commits	https://github.com/bbulb/trawl/commits/main/
	https://github.com/bbulb/trawl/commits/main/
.github/workflows	https://github.com/bbulb/trawl/tree/main/.github/workflows
.github/workflows	https://github.com/bbulb/trawl/tree/main/.github/workflows
benchmarks	https://github.com/bbulb/trawl/tree/main/benchmarks
benchmarks	https://github.com/bbulb/trawl/tree/main/benchmarks
docs	https://github.com/bbulb/trawl/tree/main/docs
docs	https://github.com/bbulb/trawl/tree/main/docs
examples	https://github.com/bbulb/trawl/tree/main/examples
examples	https://github.com/bbulb/trawl/tree/main/examples
scripts	https://github.com/bbulb/trawl/tree/main/scripts
scripts	https://github.com/bbulb/trawl/tree/main/scripts
src	https://github.com/bbulb/trawl/tree/main/src
src	https://github.com/bbulb/trawl/tree/main/src
tests	https://github.com/bbulb/trawl/tree/main/tests
tests	https://github.com/bbulb/trawl/tree/main/tests
.dockerignore	https://github.com/bbulb/trawl/blob/main/.dockerignore
.dockerignore	https://github.com/bbulb/trawl/blob/main/.dockerignore
.env.example	https://github.com/bbulb/trawl/blob/main/.env.example
.env.example	https://github.com/bbulb/trawl/blob/main/.env.example
.gitignore	https://github.com/bbulb/trawl/blob/main/.gitignore
.gitignore	https://github.com/bbulb/trawl/blob/main/.gitignore
AGENTS.md	https://github.com/bbulb/trawl/blob/main/AGENTS.md
AGENTS.md	https://github.com/bbulb/trawl/blob/main/AGENTS.md
ARCHITECTURE.md	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md
ARCHITECTURE.md	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md
CHANGELOG.md	https://github.com/bbulb/trawl/blob/main/CHANGELOG.md
CHANGELOG.md	https://github.com/bbulb/trawl/blob/main/CHANGELOG.md
CLAUDE.md	https://github.com/bbulb/trawl/blob/main/CLAUDE.md
CLAUDE.md	https://github.com/bbulb/trawl/blob/main/CLAUDE.md
CONTRIBUTING.md	https://github.com/bbulb/trawl/blob/main/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/bbulb/trawl/blob/main/CONTRIBUTING.md
Dockerfile	https://github.com/bbulb/trawl/blob/main/Dockerfile
Dockerfile	https://github.com/bbulb/trawl/blob/main/Dockerfile
LICENSE	https://github.com/bbulb/trawl/blob/main/LICENSE
LICENSE	https://github.com/bbulb/trawl/blob/main/LICENSE
README.md	https://github.com/bbulb/trawl/blob/main/README.md
README.md	https://github.com/bbulb/trawl/blob/main/README.md
docker-compose.yml	https://github.com/bbulb/trawl/blob/main/docker-compose.yml
docker-compose.yml	https://github.com/bbulb/trawl/blob/main/docker-compose.yml
environment.yml	https://github.com/bbulb/trawl/blob/main/environment.yml
environment.yml	https://github.com/bbulb/trawl/blob/main/environment.yml
pyproject.toml	https://github.com/bbulb/trawl/blob/main/pyproject.toml
pyproject.toml	https://github.com/bbulb/trawl/blob/main/pyproject.toml
README	https://github.com/bbulb/trawl
Contributing	https://github.com/bbulb/trawl
MIT license	https://github.com/bbulb/trawl
	https://github.com/bbulb/trawl#trawl
	https://github.com/bbulb/trawl/actions/workflows/ci.yml
	https://www.python.org/downloads/
	https://github.com/bbulb/trawl/blob/main/LICENSE
	https://github.com/bbulb/trawl#why-trawl
	https://github.com/bbulb/trawl#benchmark-vs-jina-reader-12-cases
	https://github.com/bbulb/trawl#external-wcxb-dev-1497-pages
WCXB	https://github.com/Murrough-Foley/web-content-extraction-benchmark
benchmarks/wcxb/README.md	https://github.com/bbulb/trawl/blob/main/benchmarks/wcxb/README.md
	https://github.com/bbulb/trawl#when-not-to-use-trawl
	https://github.com/bbulb/trawl#whats-in-the-box
	https://github.com/bbulb/trawl#project-layout
ARCHITECTURE.md	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md
	https://github.com/bbulb/trawl#requirements
llama-server	https://github.com/ggerganov/llama.cpp/tree/master/tools/server
	https://github.com/bbulb/trawl#reference-llama-server-commands
	https://github.com/bbulb/trawl#install
	https://github.com/bbulb/trawl#runtime-health-check
	https://github.com/bbulb/trawl#usage
	https://github.com/bbulb/trawl#as-a-python-library
	https://github.com/bbulb/trawl#as-an-mcp-server-stdio
	https://github.com/bbulb/trawl#as-an-http-service-docker-compose
.env.example	https://github.com/bbulb/trawl/blob/main/.env.example
	https://github.com/bbulb/trawl#wiring-into-a-client
examples/	https://github.com/bbulb/trawl/blob/main/examples
examples/claude_code_config.json	https://github.com/bbulb/trawl/blob/main/examples/claude_code_config.json
examples/mcp_gateway_config.yaml	https://github.com/bbulb/trawl/blob/main/examples/mcp_gateway_config.yaml
	https://github.com/bbulb/trawl#configuration
.env.example	https://github.com/bbulb/trawl/blob/main/.env.example
ARCHITECTURE.md#why-is-hyde-off-by-default	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md#why-is-hyde-off-by-default
	https://github.com/bbulb/trawl#testing
CONTRIBUTING.md	https://github.com/bbulb/trawl/blob/main/CONTRIBUTING.md
	https://github.com/bbulb/trawl#known-limitations
ARCHITECTURE.md#known-limitations	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md#known-limitations
	https://github.com/bbulb/trawl#documentation
ARCHITECTURE.md	https://github.com/bbulb/trawl/blob/main/ARCHITECTURE.md
CONTRIBUTING.md	https://github.com/bbulb/trawl/blob/main/CONTRIBUTING.md
CHANGELOG.md	https://github.com/bbulb/trawl/blob/main/CHANGELOG.md
CLAUDE.md	https://github.com/bbulb/trawl/blob/main/CLAUDE.md
	https://github.com/bbulb/trawl#license
LICENSE	https://github.com/bbulb/trawl/blob/main/LICENSE
ai-agents	https://github.com/topics/ai-agents
bge-m3	https://github.com/topics/bge-m3
content-extraction	https://github.com/topics/content-extraction
embeddings	https://github.com/topics/embeddings
llm	https://github.com/topics/llm
mcp	https://github.com/topics/mcp
mcp-server	https://github.com/topics/mcp-server
playwright	https://github.com/topics/playwright
python	https://github.com/topics/python
rag	https://github.com/topics/rag
trafilatura	https://github.com/topics/trafilatura
web-scraping	https://github.com/topics/web-scraping
Readme	https://github.com/bbulb/trawl#readme-ov-file
MIT license	https://github.com/bbulb/trawl#MIT-1-ov-file
Contributing	https://github.com/bbulb/trawl#contributing-ov-file
Activity	https://github.com/bbulb/trawl/activity
Custom properties	https://github.com/bbulb/trawl/custom-properties
0 forks	https://github.com/bbulb/trawl/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fbbulb%2Ftrawl&report=bbulb+%28user%29
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.