René's URL Explorer Experiment

Title: GitHub - lightfeed/extractor: Use LLMs to robustly extract web data · GitHub

Open Graph Title: GitHub - lightfeed/extractor: Use LLMs to robustly extract web data

X Title: GitHub - lightfeed/extractor: Use LLMs to robustly extract web data

Description: Use LLMs to robustly extract web data. Contribute to lightfeed/extractor development by creating an account on GitHub.

Open Graph Description: Use LLMs to robustly extract web data. Contribute to lightfeed/extractor development by creating an account on GitHub.

X Description: Use LLMs to robustly extract web data. Contribute to lightfeed/extractor development by creating an account on GitHub.

Mail addresses
support@lightfeed.ai

Opengraph URL: https://github.com/lightfeed/extractor

X: @github

direct link

Domain: github.com

react-profiling	0
react-import-map	react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:34bcf5bd-1699-7cc6-28c6-10a98e653ba3
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	AD02:2CAFAB:DC7369:123EC11:6AEBF811
html-safe-nonce	00b7b288d4250a4643e3a1407aff40a680b89e2cc0e7de87b152cceb6a72c75c
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRDAyOjJDQUZBQjpEQzczNjk6MTIzRUMxMTo2QUVCRjgxMSIsInZpc2l0b3JfaWQiOiI1MjAwMzA5NTY5MTE1ODkzOTMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac	6e4710ee79ad14c78149d1fb3a10bb7eb90315864dcf74cf659834ea9d43691c
hovercard-subject-tag	repository:976790148
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/lightfeed/extractor
twitter:image	https://opengraph.githubassets.com/ec88a64a61bd5a02a03f4bca2b23f2ba0faf86486a3de5629b66d229bcc0c5d8/lightfeed/extractor
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/ec88a64a61bd5a02a03f4bca2b23f2ba0faf86486a3de5629b66d229bcc0c5d8/lightfeed/extractor
og:image:alt	Use LLMs to robustly extract web data. Contribute to lightfeed/extractor development by creating an account on GitHub.
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	d063d52022f5f270ef55a70761d126c8085d492d3c251084b07781784c02dad4
turbo-cache-control	no-cache
go-import	github.com/lightfeed/extractor git https://github.com/lightfeed/extractor.git
octolytics-dimension-user_id	168489484
octolytics-dimension-user_login	lightfeed
octolytics-dimension-repository_id	976790148
octolytics-dimension-repository_nwo	lightfeed/extractor
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	false
octolytics-dimension-repository_network_root_id	976790148
octolytics-dimension-repository_network_root_nwo	lightfeed/extractor
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	56e59340194011ba3cb6809acfa979ccbd174288
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/lightfeed/extractor#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flightfeed%2Fextractor
GitHubRepository hosting, code review, and collaboration	https://github.com/platform
CopilotAI coding agent for tasks, projects and automated work	https://github.com/features/copilot
ActionsCI/CD and automation	https://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcement	https://github.com/features/code-quality
CodespacesCloud development environments	https://github.com/features/codespaces
Code SecurityVulnerability detection and remediation	https://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak prevention	https://github.com/security/advanced-security/secret-protection
All Features	https://github.com/features
Documentation	https://docs.github.com/en
Training	https://learn.github.com/skills
Certifications	https://learn.github.com/credentials
Build with Copilot	https://github.com/features/ai
Customize Copilot	https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrations	https://github.com/marketplace
Trending repos	https://github.com/trending
GitHub Sponsors	https://github.com/open-source/sponsors
Developer articles	https://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.	https://github.blog/changelog/
Enterprises	https://github.com/enterprise
Team	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
See all industries	https://github.com/solutions/industry
Why GitHub	https://github.com/why-github
Customer Stories	https://github.com/customer-stories
Business insights	https://github.com/solutions/executive-insights
Trust Center	https://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code security	https://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/enterprise/premium-support
Documentation	https://docs.github.com/en
Resource center	https://github.com/resources
Blog	https://github.blog/
Events & webinars	https://github.com/resources/events
Community Forum	https://github.com/orgs/community/discussions
Customer Stories	https://github.com/customer-stories
Integrations	https://github.com/marketplace
Customer support	https://help.github.com/
Training	https://learn.github.com/skills
GitHub Status	https://www.githubstatus.com/
About	https://github.com/about
Partners	https://github.com/partners
Changelog	https://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.	https://githubuniverse.com/
Pricing	https://github.com/pricing
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flightfeed%2Fextractor
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=lightfeed%2Fextractor
Reload	https://github.com/lightfeed/extractor
Reload	https://github.com/lightfeed/extractor
Reload	https://github.com/lightfeed/extractor
lightfeed	https://github.com/lightfeed
extractor	https://github.com/lightfeed/extractor
Code	https://github.com/lightfeed/extractor
Issues	https://github.com/lightfeed/extractor/issues
Pull requests2 (2)	https://github.com/lightfeed/extractor/pulls
Actions	https://github.com/lightfeed/extractor/actions
Projects	https://github.com/lightfeed/extractor/projects
Security and quality	https://github.com/lightfeed/extractor/security
Insights	https://github.com/lightfeed/extractor/pulse
	https://github.com/lightfeed/extractor
extractor	https://github.com/lightfeed/extractor
Notifications	https://github.com/login?return_to=%2Flightfeed%2Fextractor
Fork10 (10)	https://github.com/login?return_to=%2Flightfeed%2Fextractor
Star321 (321)	https://github.com/login?return_to=%2Flightfeed%2Fextractor
lightfeed.ai	https://lightfeed.ai
ai-agents	https://github.com/topics/ai-agents
article-extractor	https://github.com/topics/article-extractor
crawler	https://github.com/topics/crawler
data-engineering	https://github.com/topics/data-engineering
data-pipeline	https://github.com/topics/data-pipeline
ecommerce-scraping	https://github.com/topics/ecommerce-scraping
etl	https://github.com/topics/etl
google-gemini	https://github.com/topics/google-gemini
html-parser	https://github.com/topics/html-parser
html-to-markdown	https://github.com/topics/html-to-markdown
llm	https://github.com/topics/llm
llm-extraction	https://github.com/topics/llm-extraction
llm-scraper	https://github.com/topics/llm-scraper
markdown	https://github.com/topics/markdown
nlp	https://github.com/topics/nlp
openai	https://github.com/topics/openai
rag	https://github.com/topics/rag
web-data-extraction	https://github.com/topics/web-data-extraction
web-extraction	https://github.com/topics/web-extraction
webscraping	https://github.com/topics/webscraping
Readme	https://github.com/lightfeed/extractor#readme-ov-file
Apache-2.0 license	https://github.com/lightfeed/extractor#Apache-2.0-1-ov-file
Contributing	https://github.com/lightfeed/extractor#contributing-ov-file
Activity	https://github.com/lightfeed/extractor/activity
Custom properties	https://github.com/lightfeed/extractor/custom-properties
10 forks	https://github.com/lightfeed/extractor/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Flightfeed%2Fextractor&report=lightfeed+%28user%29
Branches	https://github.com/lightfeed/extractor/branches
Tags	https://github.com/lightfeed/extractor/tags
	https://github.com/lightfeed/extractor/branches
	https://github.com/lightfeed/extractor/tags
76 Commits	https://github.com/lightfeed/extractor/commits/main/
	https://github.com/lightfeed/extractor/commits/main/
.github/workflows	https://github.com/lightfeed/extractor/tree/main/.github/workflows
.github/workflows	https://github.com/lightfeed/extractor/tree/main/.github/workflows
src	https://github.com/lightfeed/extractor/tree/main/src
src	https://github.com/lightfeed/extractor/tree/main/src
test-data @ 6438285	https://github.com/lightfeed/extractor-test-data/tree/643828537ea05687c8e4c03985157ade3c177cb5
test-data @ 6438285	https://github.com/lightfeed/extractor-test-data/tree/643828537ea05687c8e4c03985157ade3c177cb5
tests	https://github.com/lightfeed/extractor/tree/main/tests
tests	https://github.com/lightfeed/extractor/tree/main/tests
.env.example	https://github.com/lightfeed/extractor/blob/main/.env.example
.env.example	https://github.com/lightfeed/extractor/blob/main/.env.example
.gitignore	https://github.com/lightfeed/extractor/blob/main/.gitignore
.gitignore	https://github.com/lightfeed/extractor/blob/main/.gitignore
.gitmodules	https://github.com/lightfeed/extractor/blob/main/.gitmodules
.gitmodules	https://github.com/lightfeed/extractor/blob/main/.gitmodules
.npmignore	https://github.com/lightfeed/extractor/blob/main/.npmignore
.npmignore	https://github.com/lightfeed/extractor/blob/main/.npmignore
CONTRIBUTING.md	https://github.com/lightfeed/extractor/blob/main/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/lightfeed/extractor/blob/main/CONTRIBUTING.md
LICENSE	https://github.com/lightfeed/extractor/blob/main/LICENSE
LICENSE	https://github.com/lightfeed/extractor/blob/main/LICENSE
README.md	https://github.com/lightfeed/extractor/blob/main/README.md
README.md	https://github.com/lightfeed/extractor/blob/main/README.md
jest.config.js	https://github.com/lightfeed/extractor/blob/main/jest.config.js
jest.config.js	https://github.com/lightfeed/extractor/blob/main/jest.config.js
package-lock.json	https://github.com/lightfeed/extractor/blob/main/package-lock.json
package-lock.json	https://github.com/lightfeed/extractor/blob/main/package-lock.json
package.json	https://github.com/lightfeed/extractor/blob/main/package.json
package.json	https://github.com/lightfeed/extractor/blob/main/package.json
tsconfig.json	https://github.com/lightfeed/extractor/blob/main/tsconfig.json
tsconfig.json	https://github.com/lightfeed/extractor/blob/main/tsconfig.json
README	https://github.com/lightfeed/extractor
Contributing	https://github.com/lightfeed/extractor
Apache-2.0 license	https://github.com/lightfeed/extractor
	https://camo.githubusercontent.com/e978d00a9ce25cf35afc86e7d8d475a89cce0e60560988aec98367fef882bce3/68747470733a2f2f7777772e6c69676874666565642e61692f7374617469632f736974652f6c6f676f2e737667
	https://github.com/lightfeed/extractor#----lightfeed-extractor
	https://www.npmjs.com/package/@lightfeed/extractor
	https://github.com/lightfeed/extractor/actions/workflows/test.yml
	https://github.com/lightfeed/extractor/blob/main/LICENSE
	https://www.linkedin.com/company/lightfeed-ai
	https://github.com/lightfeed/extractor#overview
	https://github.com/lightfeed/extractor#features
LLM-ready Markdown	https://github.com/lightfeed/extractor#html-to-markdown-conversion
LLM Extraction	https://github.com/lightfeed/extractor#llm-extraction-function
JSON Recovery	https://github.com/lightfeed/extractor#json-recovery
URL Validation	https://github.com/lightfeed/extractor#url-validation
Works with Playwright	https://github.com/lightfeed/extractor#using-with-playwright
AI Browser Navigation	https://github.com/lightfeed/extractor#using-with-browser-agent
@lightfeed/browser-agent	https://github.com/lightfeed/browser-agent
lightfeed.ai	https://lightfeed.ai
	https://github.com/lightfeed/extractor#installation
	https://github.com/lightfeed/extractor#usage
	https://github.com/lightfeed/extractor#e-commerce-product-extraction
testBrowserExtraction.ts	https://github.com/lightfeed/extractor/blob/main/src/dev/testBrowserExtraction.ts
	https://github.com/lightfeed/extractor#using-with-browser-agent
@lightfeed/browser-agent	https://github.com/lightfeed/browser-agent
browser-agent docs	https://github.com/lightfeed/browser-agent
	https://github.com/lightfeed/extractor#extracting-from-markdown-or-plain-text
LangChain chat model	https://js.langchain.com/docs/integrations/chat/
	https://github.com/lightfeed/extractor#custom-extraction-prompts
	https://github.com/lightfeed/extractor#extraction-context
	https://github.com/lightfeed/extractor#using-any-llm-via-langchain
LangChain-supported provider	https://js.langchain.com/docs/integrations/chat/
	https://github.com/lightfeed/extractor#managing-token-limits
	https://github.com/lightfeed/extractor#extracting-from-main-html
	https://github.com/lightfeed/extractor#extracting-images-from-html
	https://github.com/lightfeed/extractor#url-cleaning
	https://github.com/lightfeed/extractor#llm-extraction-function
	https://github.com/lightfeed/extractor#llm-configuration
LangChain chat model	https://js.langchain.com/docs/integrations/chat/
	https://github.com/lightfeed/extractor#extracttoptions-extractoroptionst-promiseextractorresultt
	https://github.com/lightfeed/extractor#options
LangChain chat model	https://js.langchain.com/docs/integrations/chat/
see below	https://github.com/lightfeed/extractor#htmlextractionoptions
	https://github.com/lightfeed/extractor#htmlextractionoptions
	https://github.com/lightfeed/extractor#return-value
	https://github.com/lightfeed/extractor#using-with-playwright
Playwright	https://playwright.dev/
	https://github.com/lightfeed/extractor#html-to-markdown-conversion
	https://github.com/lightfeed/extractor#parameters
HTML Extraction Options	https://github.com/lightfeed/extractor#html-extraction-options
	https://github.com/lightfeed/extractor#return-value-1
	https://github.com/lightfeed/extractor#example
	https://github.com/lightfeed/extractor#json-recovery
	https://github.com/lightfeed/extractor#url-validation
	https://github.com/lightfeed/extractor#how-url-validation-works
	https://github.com/lightfeed/extractor#development
	https://github.com/lightfeed/extractor#setup
	https://github.com/lightfeed/extractor#scripts
	https://github.com/lightfeed/extractor#running-local-tests
	https://github.com/lightfeed/extractor#testing
	https://github.com/lightfeed/extractor#html-to-markdown-integration-tests
	https://github.com/lightfeed/extractor#running-specific-tests
	https://github.com/lightfeed/extractor#support
	https://github.com/lightfeed/extractor#license
lightfeed.ai	https://lightfeed.ai
ai-agents	https://github.com/topics/ai-agents
article-extractor	https://github.com/topics/article-extractor
crawler	https://github.com/topics/crawler
data-engineering	https://github.com/topics/data-engineering
data-pipeline	https://github.com/topics/data-pipeline
ecommerce-scraping	https://github.com/topics/ecommerce-scraping
etl	https://github.com/topics/etl
google-gemini	https://github.com/topics/google-gemini
html-parser	https://github.com/topics/html-parser
html-to-markdown	https://github.com/topics/html-to-markdown
llm	https://github.com/topics/llm
llm-extraction	https://github.com/topics/llm-extraction
llm-scraper	https://github.com/topics/llm-scraper
markdown	https://github.com/topics/markdown
nlp	https://github.com/topics/nlp
openai	https://github.com/topics/openai
rag	https://github.com/topics/rag
web-data-extraction	https://github.com/topics/web-data-extraction
web-extraction	https://github.com/topics/web-extraction
webscraping	https://github.com/topics/webscraping
Readme	https://github.com/lightfeed/extractor#readme-ov-file
Apache-2.0 license	https://github.com/lightfeed/extractor#Apache-2.0-1-ov-file
Contributing	https://github.com/lightfeed/extractor#contributing-ov-file
Activity	https://github.com/lightfeed/extractor/activity
Custom properties	https://github.com/lightfeed/extractor/custom-properties
10 forks	https://github.com/lightfeed/extractor/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Flightfeed%2Fextractor&report=lightfeed+%28user%29
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.