René's URL Explorer Experiment

{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://urlpipe.dev/#organization","name":"URLpipe","url":"https://urlpipe.dev","logo":"https://urlpipe.dev/icon.png","sameAs":["https://github.com/URLpipe"],"founder":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"}},{"@type":"WebSite","@id":"https://urlpipe.dev/#website","url":"https://urlpipe.dev","name":"URLpipe","publisher":{"@id":"https://urlpipe.dev/#organization"}}]}

{"@context":"https://schema.org","@graph":[{"@type":"TechArticle","@id":"https://urlpipe.dev/code/python/rendered-html#article","headline":"Get JavaScript-rendered HTML in Python — working code","description":"Fetch the HTML of a JavaScript page after it rendered, in Python: sync, async with a webhook, and error handling. Tested code.","url":"https://urlpipe.dev/code/python/rendered-html","mainEntityOfPage":"https://urlpipe.dev/code/python/rendered-html","datePublished":"2026-09-24","dateModified":"2026-09-24","author":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"},"publisher":{"@id":"https://urlpipe.dev/#organization"},"image":"https://urlpipe.dev/og-image.png"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How is this different from fetching the URL myself?","acceptedAnswer":{"@type":"Answer","text":"A plain GET returns what the server sent before any JavaScript ran. /html returns the DOM after the page's scripts ran in real Chrome — the HTML a visitor's browser actually holds."}},{"@type":"Question","name":"Can I wait for content that loads late?","acceptedAnswer":{"@type":"Answer","text":"Yes: page_options.wait_for_selector waits up to 10 seconds for an element to appear, and delay adds a fixed wait. If the element never appears the request is a 422 and costs nothing."}},{"@type":"Question","name":"Does URLpipe follow robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Yes, by default, for every project. You can turn it off per project, and then you are responsible for having the right to fetch those pages."}},{"@type":"Question","name":"Do I need an SDK to call URLpipe from Python?","acceptedAnswer":{"@type":"Answer","text":"No SDK is needed: requests is the only dependency, because the API is one POST per job. If you'd rather use one, the official client installs with pip install urlpipe and adds retries that never bill twice, typed errors and webhook verification."}}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://urlpipe.dev/"},{"@type":"ListItem","position":2,"name":"Code recipes","item":"https://urlpipe.dev/code"},{"@type":"ListItem","position":3,"name":"Python","item":"https://urlpipe.dev/code/python"},{"@type":"ListItem","position":4,"name":"Get the rendered HTML of a JavaScript page in Python","item":"https://urlpipe.dev/code/python/rendered-html"}]},{"@type":"HowTo","name":"Get the rendered HTML of a JavaScript page in Python","description":"To get the HTML of a JavaScript-rendered page in Python, POST its URL to https://urlpipe.dev/html with sync set to true. The body is the DOM after the page's scripts ran in real Chrome, serialized as HTML — what a visitor's browser holds, not what the server first sent — read with res.text, for 1 credit.","url":"https://urlpipe.dev/code/python/rendered-html","tool":[{"@type":"HowToTool","name":"Python 3.9+"}],"step":[{"@type":"HowToStep","name":"Before you start","text":"One dependency, requests. Put your API key in the environment so it never lands in the file.","url":"https://urlpipe.dev/code/python/rendered-html#setup"},{"@type":"HowToStep","name":"Save the rendered HTML and report its size","text":""sync": True holds the connection open until the result is ready and answers with it. Pass timeout= every time: requests has no default, and a page that never finishes loading would otherwise hang your process. Write res.content (bytes) rather than res.text, so the file is byte-for-byte what came back and its size is the real size.","url":"https://urlpipe.dev/code/python/rendered-html#sync"},{"@type":"HowToStep","name":"The async variant: a token, a webhook and a poll","text":"Leave out sync and the answer is a token, straight away. for … else is the idiom for "ran out of attempts": the else runs only when the loop never hit break.","url":"https://urlpipe.dev/code/python/rendered-html#async"},{"@type":"HowToStep","name":"Handle errors and retries","text":"raise_for_status() is fine for a script; a service wants to tell the failures apart. Check the status before calling res.json() — an error from a proxy in front of the API may not be JSON. sys.exit(message) prints to stderr and exits 1.","url":"https://urlpipe.dev/code/python/rendered-html#errors"}]}]}