René's URL Explorer Experiment

{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://urlpipe.dev/#organization","name":"URLpipe","url":"https://urlpipe.dev","logo":"https://urlpipe.dev/icon.png","sameAs":["https://github.com/URLpipe"],"founder":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"}},{"@type":"WebSite","@id":"https://urlpipe.dev/#website","url":"https://urlpipe.dev","name":"URLpipe","publisher":{"@id":"https://urlpipe.dev/#organization"}}]}

{"@context":"https://schema.org","@graph":[{"@type":"TechArticle","@id":"https://urlpipe.dev/code/python/html-to-markdown#article","headline":"Convert a URL to Markdown in Python — working code","description":"Turn any URL into clean Markdown in Python, rendered in real Chrome first: sync, async with a webhook, and retries. Tested code.","url":"https://urlpipe.dev/code/python/html-to-markdown","mainEntityOfPage":"https://urlpipe.dev/code/python/html-to-markdown","datePublished":"2026-09-24","dateModified":"2026-09-24","author":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"},"publisher":{"@id":"https://urlpipe.dev/#organization"},"image":"https://urlpipe.dev/og-image.png"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Does the Markdown conversion use an LLM?","acceptedAnswer":{"@type":"Answer","text":"No. It is a deterministic walk over the rendered DOM, so it is fast, costs 1 credit a page, and gives the same output for the same page. Only /summarize, /keywords, and /extract call a model."}},{"@type":"Question","name":"Does it work on pages built with JavaScript?","acceptedAnswer":{"@type":"Answer","text":"Yes. The page is loaded in real Chrome and its scripts run before the DOM is read, so React, Vue and other client-rendered pages convert like static ones."}},{"@type":"Question","name":"How do I keep the Markdown for later?","acceptedAnswer":{"@type":"Answer","text":"Write the body to a .md file, or rely on the cache: the same URL within max_age (7 days by default) is answered from storage for free, and any result can be fetched again by its token for 30 days."}},{"@type":"Question","name":"Do I need an SDK to call URLpipe from Python?","acceptedAnswer":{"@type":"Answer","text":"No SDK is needed: requests is the only dependency, because the API is one POST per job. If you'd rather use one, the official client installs with pip install urlpipe and adds retries that never bill twice, typed errors and webhook verification."}}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://urlpipe.dev/"},{"@type":"ListItem","position":2,"name":"Code recipes","item":"https://urlpipe.dev/code"},{"@type":"ListItem","position":3,"name":"Python","item":"https://urlpipe.dev/code/python"},{"@type":"ListItem","position":4,"name":"Convert a web page to Markdown in Python","item":"https://urlpipe.dev/code/python/html-to-markdown"}]},{"@type":"HowTo","name":"Convert a web page to Markdown in Python","description":"To convert a web page to Markdown in Python, POST its URL to https://urlpipe.dev/markdown with sync set to true; the response body is the Markdown, read with res.text. The page is rendered in real Chrome first, then converted by a deterministic walk over the DOM, not a model, for 1 credit a page.","url":"https://urlpipe.dev/code/python/html-to-markdown","tool":[{"@type":"HowToTool","name":"Python 3.9+"}],"step":[{"@type":"HowToStep","name":"Before you start","text":"One dependency, requests. Put your API key in the environment so it never lands in the file.","url":"https://urlpipe.dev/code/python/html-to-markdown#setup"},{"@type":"HowToStep","name":"Print a page as Markdown","text":""sync": True holds the connection open until the result is ready and answers with it. Pass timeout= every time: requests has no default, and a page that never finishes loading would otherwise hang your process. The body is the Markdown itself, so res.text is the whole job.","url":"https://urlpipe.dev/code/python/html-to-markdown#sync"},{"@type":"HowToStep","name":"The async variant: a token, a webhook and a poll","text":"Leave out sync and the answer is a token, straight away. for … else is the idiom for "ran out of attempts": the else runs only when the loop never hit break.","url":"https://urlpipe.dev/code/python/html-to-markdown#async"},{"@type":"HowToStep","name":"Handle errors and retries","text":"raise_for_status() is fine for a script; a service wants to tell the failures apart. Check the status before calling res.json() — an error from a proxy in front of the API may not be JSON. sys.exit(message) prints to stderr and exits 1.","url":"https://urlpipe.dev/code/python/html-to-markdown#errors"}]}]}