René's URL Explorer Experiment


Title: Examples - Reader Docs

Open Graph Title: Reader - Open-source web scraping for LLMs

X Title: Reader - Open-source web scraping for LLMs

Description: Runnable code examples for every feature of Reader.

Open Graph Description: Production-grade web scraping engine. Two primitives. Clean markdown. Ready for your agents.

X Description: Production-grade web scraping engine. Two primitives. Clean markdown. Ready for your agents.

Keywords:

Opengraph URL: https://reader.dev

direct link

Domain: reader.dev

authorVakra Dev
og:site_nameReader
og:imagehttps://reader.dev/og-image.png
og:typewebsite
twitter:cardsummary_large_image
twitter:imagehttps://reader.dev/og-image.png

Links:

https://reader.dev/
Markdownhttps://reader.dev/markdown
Docshttps://reader.dev/docs
https://github.com/vakra-dev/reader
Introductionhttps://reader.dev/docs/getting-started/introduction
Installationhttps://reader.dev/docs/getting-started/installation
Quickstarthttps://reader.dev/docs/getting-started/quickstart
Exampleshttps://reader.dev/docs/getting-started/examples
Scrapinghttps://reader.dev/docs/concepts/scraping
Crawlinghttps://reader.dev/docs/concepts/crawling
Content Extractionhttps://reader.dev/docs/concepts/content-extraction
Browser Poolhttps://reader.dev/docs/concepts/browser-pool
Browser Sessionshttps://reader.dev/docs/concepts/browser-sessions
Engine Waterfallhttps://reader.dev/docs/concepts/engine-waterfall
Proxy Tiershttps://reader.dev/docs/concepts/proxy-tiers
Error Handlinghttps://reader.dev/docs/concepts/error-handling
Basic Scrapinghttps://reader.dev/docs/guides/basic-scraping
Batch Scrapinghttps://reader.dev/docs/guides/batch-scraping
Website Crawlinghttps://reader.dev/docs/guides/website-crawling
Proxy Configurationhttps://reader.dev/docs/guides/proxy-configuration
CLIhttps://reader.dev/docs/guides/cli
Daemon Modehttps://reader.dev/docs/guides/daemon-mode
Browser Sessionshttps://reader.dev/docs/guides/browser-sessions
Deploymenthttps://reader.dev/docs/guides/deployment
ReaderClienthttps://reader.dev/docs/api-reference/reader-client
scrape()https://reader.dev/docs/api-reference/scrape
crawl()https://reader.dev/docs/api-reference/crawl
ScrapeOptionshttps://reader.dev/docs/api-reference/scrape-options
ScrapeResulthttps://reader.dev/docs/api-reference/scrape-result
CrawlOptionshttps://reader.dev/docs/api-reference/crawl-options
CrawlResulthttps://reader.dev/docs/api-reference/crawl-result
BrowserSessionhttps://reader.dev/docs/api-reference/browser-session
Errorshttps://reader.dev/docs/api-reference/errors
reader repositoryhttps://github.com/vakra-dev/reader/tree/main/examples
Basic scrapeScrape a single URL and print the markdown result.https://github.com/vakra-dev/reader/blob/main/examples/basic/basic-scrape.ts
All formatsRequest markdown and HTML in one call.https://github.com/vakra-dev/reader/blob/main/examples/basic/all-formats.ts
Batch scrapeScrape multiple URLs in parallel with progress tracking.https://github.com/vakra-dev/reader/blob/main/examples/basic/batch-scrape.ts
Large batch scrapeScale up concurrency and tune the pool for hundreds of URLs.https://github.com/vakra-dev/reader/blob/main/examples/basic/large-batch-scrape.ts
Crawl a websiteDiscover and scrape every page on a domain.https://github.com/vakra-dev/reader/blob/main/examples/basic/crawl-website.ts
Browser pool configTune pool size, recycling, and health checks.https://github.com/vakra-dev/reader/blob/main/examples/basic/browser-pool-config.ts
Single proxyPass a single proxy per request.https://github.com/vakra-dev/reader/blob/main/examples/basic/with-proxy.ts
Proxy pool rotationRotate through a pool with round-robin or random strategy.https://github.com/vakra-dev/reader/blob/main/examples/basic/proxy-pool.ts
Cloudflare bypassScrape sites behind Cloudflare challenges.https://github.com/vakra-dev/reader/blob/main/examples/basic/cloudflare-bypass.ts
Anthropic summaryPipe scraped markdown to Claude for summarization.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/anthropic-summary.ts
OpenAI summarySame pattern with GPT-4 or GPT-4o-mini.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/openai-summary.ts
Vercel AI streamingStream scrape + LLM output with the Vercel AI SDK.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/vercel-ai-stream.ts
LangChain loaderUse Reader as a LangChain document loader.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/langchain-loader.ts
LlamaIndex loaderUse Reader as a LlamaIndex data connector.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/llamaindex-loader.ts
Pinecone ingestScrape, chunk, embed, and upsert to Pinecone.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/pinecone-ingest.ts
Qdrant ingestSame pattern with Qdrant as the vector store.https://github.com/vakra-dev/reader/blob/main/examples/ai-tools/qdrant-ingest.ts
Express serverEmbed Reader in an HTTP API with a persistent client.https://github.com/vakra-dev/reader/tree/main/examples/production/express-server
BullMQ job queueProcess scrape jobs from a Redis-backed queue with retries.https://github.com/vakra-dev/reader/tree/main/examples/production/job-queue-bullmq
Browser pool scalingPatterns for scaling the pool under real load.https://github.com/vakra-dev/reader/tree/main/examples/production/browser-pool-scaling
Docker ComposeProduction Dockerfile + compose with correct shm_size and capabilities.https://github.com/vakra-dev/reader/tree/main/examples/deployment/docker
AWS LambdaDeploy Reader as a serverless function handler.https://github.com/vakra-dev/reader/tree/main/examples/deployment/aws-lambda
Vercel FunctionsRun Reader from a Vercel Edge/Node function.https://github.com/vakra-dev/reader/tree/main/examples/deployment/vercel-functions
QuickstartWalk through your first scrape in more depth.https://reader.dev/docs/getting-started/quickstart
API ReferenceFull type reference for every option.https://reader.dev/docs/api-reference/reader-client

Viewport: width=device-width, initial-scale=1

URLs of crawlers that visited me.