René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Performance bottleneck in `prune_unwanted_nodes` causing 200ms per call","articleBody":"When profiling `trafilatura.bare_extraction` method for some pages that took us a while to parse, I found that significant performance issues in `extract_content`  method. \r\n\r\n**Root cause**: Too many calls to `prune_unwanted_node` with each call taking upto ~200ms.\r\n\r\n**Steps to reproduce**:\r\n1. Run python with cProfile profiler on `bare_extraction`\r\n2. Try to scrape and parse this page https://nlewo.github.io/nixos-manual-sphinx/generated/options-db.xml.html\r\n3. Observe `prune_unwanted_node` timing\r\n\r\n**Questions**\r\n1. Are there existing configuration options or alternative methods in trafilatura to bypass/optimize this operation?\r\n2. Could we leverage Rust/C++ based HTML parsing libraries (like lxml-rs or html5ever) to accelerate node pruning?\r\n3. Could we leverage caching to reduce latency?\r\n","author":{"url":"https://github.com/thsunkid","@type":"Person","name":"thsunkid"},"datePublished":"2024-11-23T20:48:47Z","url":"https://github.com/adbar/trafilatura/issues/750"}