René's URL Explorer Experiment

{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://urlpipe.dev/#organization","name":"URLpipe","url":"https://urlpipe.dev","logo":"https://urlpipe.dev/icon.png","sameAs":["https://github.com/URLpipe"],"founder":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"}},{"@type":"WebSite","@id":"https://urlpipe.dev/#website","url":"https://urlpipe.dev","name":"URLpipe","publisher":{"@id":"https://urlpipe.dev/#organization"}}]}

{"@context":"https://schema.org","@graph":[{"@type":"TechArticle","@id":"https://urlpipe.dev/code/java/rendered-html#article","headline":"Get JavaScript-rendered HTML in Java — working code","description":"Fetch the HTML of a JavaScript page after it rendered, in Java: sync, async with a webhook, and error handling. Tested code.","url":"https://urlpipe.dev/code/java/rendered-html","mainEntityOfPage":"https://urlpipe.dev/code/java/rendered-html","datePublished":"2026-09-24","dateModified":"2026-09-24","author":{"@type":"Person","name":"Roger Campos","url":"https://rogercampos.com"},"publisher":{"@id":"https://urlpipe.dev/#organization"},"image":"https://urlpipe.dev/og-image.png"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How is this different from fetching the URL myself?","acceptedAnswer":{"@type":"Answer","text":"A plain GET returns what the server sent before any JavaScript ran. /html returns the DOM after the page's scripts ran in real Chrome — the HTML a visitor's browser actually holds."}},{"@type":"Question","name":"Can I wait for content that loads late?","acceptedAnswer":{"@type":"Answer","text":"Yes: page_options.wait_for_selector waits up to 10 seconds for an element to appear, and delay adds a fixed wait. If the element never appears the request is a 422 and costs nothing."}},{"@type":"Question","name":"Does URLpipe follow robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Yes, by default, for every project. You can turn it off per project, and then you are responsible for having the right to fetch those pages."}},{"@type":"Question","name":"Do I need an SDK to call URLpipe from Java?","acceptedAnswer":{"@type":"Answer","text":"No SDK. HttpClient is in the JDK; the programs that read JSON add Gson, one jar, because the JDK has no parser."}}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://urlpipe.dev/"},{"@type":"ListItem","position":2,"name":"Code recipes","item":"https://urlpipe.dev/code"},{"@type":"ListItem","position":3,"name":"Java","item":"https://urlpipe.dev/code/java"},{"@type":"ListItem","position":4,"name":"Get the rendered HTML of a JavaScript page in Java","item":"https://urlpipe.dev/code/java/rendered-html"}]},{"@type":"HowTo","name":"Get the rendered HTML of a JavaScript page in Java","description":"To get the HTML of a JavaScript-rendered page in Java, POST its URL to https://urlpipe.dev/html with sync set to true. The body is the DOM after the page's scripts ran in real Chrome, serialized as HTML — what a visitor's browser holds, not what the server first sent — read with HttpResponse.BodyHandlers.ofString(), for 1 credit.","url":"https://urlpipe.dev/code/java/rendered-html","tool":[{"@type":"HowToTool","name":"Java 17+"}],"step":[{"@type":"HowToStep","name":"Before you start","text":"Java 17 or later: java.net.http is in the JDK. The async and error-handling programs read JSON, and the JDK has no parser, so those use Gson — one jar, no dependencies of its own. In a Maven or Gradle build add com.google.code.gson:gson:2.13.1; to try a file on its own, download the jar next to it.","url":"https://urlpipe.dev/code/java/rendered-html#setup"},{"@type":"HowToStep","name":"Save the rendered HTML and report its size","text":""sync": true keeps the request open until the result is ready. BodyHandlers.ofString() reads the whole body as text, decoded with the charset the response names. HttpClient does not throw on a 4xx or 5xx, so check statusCode(). body.length() counts UTF-16 chars; the size on disk is the UTF-8 byte count.","url":"https://urlpipe.dev/code/java/rendered-html#sync"},{"@type":"HowToStep","name":"The async variant: a token, a webhook and a poll","text":"Leave out sync and the answer is a token, straight away. One HttpClient is shared by every call — it holds the connection pool, so build it once — and a switch expression covers the answers GET /result/:token can give.","url":"https://urlpipe.dev/code/java/rendered-html#async"},{"@type":"HowToStep","name":"Handle errors and retries","text":"A checked URLpipeException makes the caller decide what a refusal means. firstValueAsLong reads Retry-After without a parse of your own, and a body that is not JSON (a proxy in front of the API can answer in HTML) falls back to an empty object.","url":"https://urlpipe.dev/code/java/rendered-html#errors"}]}]}