René's URL Explorer Experiment


Title: Engineering notes — how URLpipe is built

Open Graph Title: Engineering notes — how URLpipe is built

X Title: Engineering notes — how URLpipe is built

Description: Notes from building a web data API: our own rendering engine, one-visit scrapes, pacing slow sites, and billing only what worked.

Open Graph Description: Notes from building a web data API: our own rendering engine, one-visit scrapes, pacing slow sites, and billing only what worked.

X Description: Notes from building a web data API: our own rendering engine, one-visit scrapes, pacing slow sites, and billing only what worked.

Mail addresses
contact@urlpipe.dev

Opengraph URL: https://urlpipe.dev/notes

direct link

Domain: urlpipe.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://urlpipe.dev/#organization","name":"URLpipe","url":"https://urlpipe.dev","logo":"https://urlpipe.dev/icon.png","sameAs":["https://github.com/URLpipe"],"founder":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"}},{"@type":"WebSite","@id":"https://urlpipe.dev/#website","url":"https://urlpipe.dev","name":"URLpipe","publisher":{"@id":"https://urlpipe.dev/#organization"}}]}
{"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","name":"Engineering notes — how URLpipe is built","description":"Notes from building a web data API: our own rendering engine, one-visit scrapes, pacing slow sites, and billing only what worked.","url":"https://urlpipe.dev/notes","hasPart":[{"@type":"WebPage","name":"Why we built our own rendering engine","url":"https://urlpipe.dev/notes/our-own-rendering-engine"},{"@type":"WebPage","name":"One page visit, several answers","url":"https://urlpipe.dev/notes/one-visit-scrape"},{"@type":"WebPage","name":"Pacing a site that can't keep up","url":"https://urlpipe.dev/notes/pacing-slow-sites"},{"@type":"WebPage","name":"Why async is the default","url":"https://urlpipe.dev/notes/async-by-default"},{"@type":"WebPage","name":"Billing only what worked","url":"https://urlpipe.dev/notes/billing-only-what-worked"},{"@type":"WebPage","name":"Request timing as marks, not spans","url":"https://urlpipe.dev/notes/request-timing-as-marks"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://urlpipe.dev/"},{"@type":"ListItem","position":2,"name":"Engineering notes","item":"https://urlpipe.dev/notes"}]}]}

view-transitionsame-origin
csrf-paramauthenticity_token
csrf-tokeng5vvFGZn2HhhEhs3FLwWVVpZWeXv15mZh_7Nz3uLxLi3unf1GNCRP9bf8Pt0tKssnaZX8m66b_rwvRBRLRSh9w
csp-nonce3b7339427ab7898273fd7987dd37e098
og:typewebsite
og:site_nameURLpipe
og:localeen_US
og:imagehttps://urlpipe.dev/og-image.png
og:image:width1200
og:image:height630
og:image:altURLpipe — URL in. Clean data out.
twitter:cardsummary_large_image
twitter:imagehttps://urlpipe.dev/og-image.png
twitter:image:altURLpipe — URL in. Clean data out.
theme-color#3b82f6

Links:

Skip to main contenthttps://urlpipe.dev/notes#main
URLpipehttps://urlpipe.dev/
Web scraping APIhttps://urlpipe.dev/web-scraping-api
URL to Markdown APIhttps://urlpipe.dev/url-to-markdown-api
MCP serverhttps://urlpipe.dev/mcp-server
Screenshot APIhttps://urlpipe.dev/screenshot-api
Lighthouse APIhttps://urlpipe.dev/lighthouse-api
Metadata APIhttps://urlpipe.dev/metadata-api
Console APIhttps://urlpipe.dev/console-api
Rendered HTML APIhttps://urlpipe.dev/rendered-html-api
URL to PDF APIhttps://urlpipe.dev/url-to-pdf-api
Website technology lookup APIhttps://urlpipe.dev/website-technology-lookup-api
Structured data extraction APIhttps://urlpipe.dev/structured-data-extraction-api
Keyword extraction APIhttps://urlpipe.dev/keyword-extraction-api
URL summarization APIhttps://urlpipe.dev/url-summarization-api
Web access for AI agentshttps://urlpipe.dev/use-cases/ai-agents
Web pages into a RAG pipelinehttps://urlpipe.dev/use-cases/rag-ingestion
Link previews and unfurlshttps://urlpipe.dev/use-cases/link-previews
Monitoring client websiteshttps://urlpipe.dev/use-cases/client-site-monitoring
Monitoring pages for changeshttps://urlpipe.dev/use-cases/content-monitoring
Archiving pages as screenshotshttps://urlpipe.dev/use-cases/visual-archiving
URL features inside a multi-tenant apphttps://urlpipe.dev/use-cases/multi-tenant-apps
Free toolshttps://urlpipe.dev/tools
Docshttps://urlpipe.dev/docs
Pricinghttps://urlpipe.dev/pricing
Guideshttps://urlpipe.dev/guides
Code recipeshttps://urlpipe.dev/code
Glossaryhttps://urlpipe.dev/glossary
Researchhttps://urlpipe.dev/research
Comparehttps://urlpipe.dev/compare
Changeloghttps://urlpipe.dev/changelog
Engineering noteshttps://urlpipe.dev/notes
Sign inhttps://urlpipe.dev/auth/session/new
Start freehttps://urlpipe.dev/auth/sign-up/new
Web scraping APIhttps://urlpipe.dev/web-scraping-api
URL to Markdown APIhttps://urlpipe.dev/url-to-markdown-api
MCP serverhttps://urlpipe.dev/mcp-server
Screenshot APIhttps://urlpipe.dev/screenshot-api
Lighthouse APIhttps://urlpipe.dev/lighthouse-api
Metadata APIhttps://urlpipe.dev/metadata-api
Console APIhttps://urlpipe.dev/console-api
Rendered HTML APIhttps://urlpipe.dev/rendered-html-api
URL to PDF APIhttps://urlpipe.dev/url-to-pdf-api
Website technology lookup APIhttps://urlpipe.dev/website-technology-lookup-api
Structured data extraction APIhttps://urlpipe.dev/structured-data-extraction-api
Keyword extraction APIhttps://urlpipe.dev/keyword-extraction-api
URL summarization APIhttps://urlpipe.dev/url-summarization-api
Web access for AI agentshttps://urlpipe.dev/use-cases/ai-agents
Web pages into a RAG pipelinehttps://urlpipe.dev/use-cases/rag-ingestion
Link previews and unfurlshttps://urlpipe.dev/use-cases/link-previews
Monitoring client websiteshttps://urlpipe.dev/use-cases/client-site-monitoring
Monitoring pages for changeshttps://urlpipe.dev/use-cases/content-monitoring
Archiving pages as screenshotshttps://urlpipe.dev/use-cases/visual-archiving
URL features inside a multi-tenant apphttps://urlpipe.dev/use-cases/multi-tenant-apps
Free toolshttps://urlpipe.dev/tools
Docshttps://urlpipe.dev/docs
Pricinghttps://urlpipe.dev/pricing
Guideshttps://urlpipe.dev/guides
Code recipeshttps://urlpipe.dev/code
Glossaryhttps://urlpipe.dev/glossary
Researchhttps://urlpipe.dev/research
Comparehttps://urlpipe.dev/compare
Changeloghttps://urlpipe.dev/changelog
Engineering noteshttps://urlpipe.dev/notes
Start freehttps://urlpipe.dev/auth/sign-up/new
Sign inhttps://urlpipe.dev/auth/session/new
Why we built our own rendering engineOne warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.Readhttps://urlpipe.dev/notes/our-own-rendering-engine
One page visit, several answersHow /scrape reads several results off one page load and stays identical to calling each endpoint on its own.Readhttps://urlpipe.dev/notes/one-visit-scrape
Pacing a site that can't keep upWhen a target site falls over under load, slow down and try again — without failing the request or billing the wait.Readhttps://urlpipe.dev/notes/pacing-slow-sites
Why async is the defaultPages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.Readhttps://urlpipe.dev/notes/async-by-default
Billing only what workedFailures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.Readhttps://urlpipe.dev/notes/billing-only-what-worked
Request timing as marks, not spansStore four moments per request and derive every duration from them — and publish exactly one number.Readhttps://urlpipe.dev/notes/request-timing-as-marks
Start freehttps://urlpipe.dev/auth/sign-up/new
URLpipehttps://urlpipe.dev/
Web scraping APIhttps://urlpipe.dev/web-scraping-api
URL to Markdown APIhttps://urlpipe.dev/url-to-markdown-api
MCP serverhttps://urlpipe.dev/mcp-server
Screenshot APIhttps://urlpipe.dev/screenshot-api
Lighthouse APIhttps://urlpipe.dev/lighthouse-api
Metadata APIhttps://urlpipe.dev/metadata-api
Console APIhttps://urlpipe.dev/console-api
Rendered HTML APIhttps://urlpipe.dev/rendered-html-api
URL to PDF APIhttps://urlpipe.dev/url-to-pdf-api
Website technology lookup APIhttps://urlpipe.dev/website-technology-lookup-api
Structured data extraction APIhttps://urlpipe.dev/structured-data-extraction-api
Keyword extraction APIhttps://urlpipe.dev/keyword-extraction-api
URL summarization APIhttps://urlpipe.dev/url-summarization-api
Free toolshttps://urlpipe.dev/tools
Pricinghttps://urlpipe.dev/pricing
Docshttps://urlpipe.dev/docs
Client librarieshttps://urlpipe.dev/docs/client-libraries
Guideshttps://urlpipe.dev/guides
Code recipeshttps://urlpipe.dev/code
Use caseshttps://urlpipe.dev/use-cases
Integrationshttps://urlpipe.dev/integrations
Glossaryhttps://urlpipe.dev/glossary
Researchhttps://urlpipe.dev/research
Engineering noteshttps://urlpipe.dev/notes
Changeloghttps://urlpipe.dev/changelog
llms.txthttps://urlpipe.dev/llms.txt
All comparisonshttps://urlpipe.dev/compare
URLpipe vs Firecrawlhttps://urlpipe.dev/compare/urlpipe-vs-firecrawl
URLpipe vs Jina Readerhttps://urlpipe.dev/compare/urlpipe-vs-jina-reader
Best HTML-to-Markdown APIshttps://urlpipe.dev/best-html-to-markdown-apis
Jina Reader alternativeshttps://urlpipe.dev/alternatives/jina-reader
Microlink alternativeshttps://urlpipe.dev/alternatives/microlink
ScreenshotOne alternativeshttps://urlpipe.dev/alternatives/screenshotone
Abouthttps://urlpipe.dev/about
Securityhttps://urlpipe.dev/security
Statushttps://urlpipe.dev/status
GitHubhttps://github.com/URLpipe
Privacyhttps://urlpipe.dev/privacy
Termshttps://urlpipe.dev/terms
Sign inhttps://urlpipe.dev/auth/session/new
Legal noticehttps://urlpipe.dev/terms#legal-notice
Roger Camposhttps://rogercampos.com

Viewport: width=device-width,initial-scale=1


URLs of crawlers that visited me.