René's URL Explorer Experiment


Title: Trafilatura: discover web content, extract text and metadata — Trafilatura 2.3.1 documentation

Open Graph Title: Trafilatura: discover web content, extract text and metadata

Description: Trafilatura is a Python package and command-line tool to gather text from the web and turn raw HTML into structured data. It handles web crawling, downloads, scraping, and extraction of main texts, metadata and comments.

Open Graph Description: Python package Python versions Code Coverage Downloads Reference DOI: 10.18653/v1/2021.acl-demo.15, Trafilatura command-line demo Description: Trafilatura is a comprehensive Python package and comm...

Opengraph URL: https://trafilatura.readthedocs.io/index.html

direct link

Domain: trafilatura.readthedocs.io

og:typewebsite
og:site_nameTrafilatura
og:imagehttps://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-logo.png
og:image:altTrafilatura
docsearch:languageen
docsearch:version2.3.1
readthedocs-project-slugtrafilatura
readthedocs-version-sluglatest
readthedocs-resolver-filename/
readthedocs-http-status200

Links:

Skip to main contenthttps://trafilatura.readthedocs.io/#main-content
https://trafilatura.readthedocs.io/
Installation https://trafilatura.readthedocs.io/installation.html
Quickstart https://trafilatura.readthedocs.io/quickstart.html
Usage https://trafilatura.readthedocs.io/usage.html
Tutorials https://trafilatura.readthedocs.io/tutorials.html
FAQ https://trafilatura.readthedocs.io/faq.html
Troubleshooting https://trafilatura.readthedocs.io/troubleshooting.html
How extraction works https://trafilatura.readthedocs.io/extraction-overview.html
Benchmarks and evaluation https://trafilatura.readthedocs.io/evaluation.html
Core functions https://trafilatura.readthedocs.io/corefunctions.html
Settings and customization https://trafilatura.readthedocs.io/settings.html
Deprecations and migration https://trafilatura.readthedocs.io/deprecations.html
Uses & citations https://trafilatura.readthedocs.io/used-by.html
Running the tests https://trafilatura.readthedocs.io/tests.html
Background https://trafilatura.readthedocs.io/background.html
Blog https://adrien.barbaresi.eu/blog/tag/trafilatura.html
GitHubhttps://github.com/adbar/trafilatura
Installation https://trafilatura.readthedocs.io/installation.html
Quickstart https://trafilatura.readthedocs.io/quickstart.html
Usage https://trafilatura.readthedocs.io/usage.html
Tutorials https://trafilatura.readthedocs.io/tutorials.html
FAQ https://trafilatura.readthedocs.io/faq.html
Troubleshooting https://trafilatura.readthedocs.io/troubleshooting.html
How extraction works https://trafilatura.readthedocs.io/extraction-overview.html
Benchmarks and evaluation https://trafilatura.readthedocs.io/evaluation.html
Core functions https://trafilatura.readthedocs.io/corefunctions.html
Settings and customization https://trafilatura.readthedocs.io/settings.html
Deprecations and migration https://trafilatura.readthedocs.io/deprecations.html
Uses & citations https://trafilatura.readthedocs.io/used-by.html
Running the tests https://trafilatura.readthedocs.io/tests.html
Background https://trafilatura.readthedocs.io/background.html
Blog https://adrien.barbaresi.eu/blog/tag/trafilatura.html
GitHubhttps://github.com/adbar/trafilatura
#https://trafilatura.readthedocs.io/#trafilatura-discover-web-content-extract-text-and-metadata
https://pypi.python.org/pypi/trafilatura
https://pypi.python.org/pypi/trafilatura
https://codecov.io/gh/adbar/trafilatura
https://pepy.tech/project/trafilatura
https://aclanthology.org/2021.acl-demo.15/
https://trafilatura.readthedocs.org/
#https://trafilatura.readthedocs.io/#description
widely usedhttps://trafilatura.readthedocs.io/used-by.html
thousands of projectshttps://github.com/adbar/trafilatura/network/dependents
#https://trafilatura.readthedocs.io/#features
XML-TEIhttps://tei-c.org/
#https://trafilatura.readthedocs.io/#evaluation
benchmark sectionhttps://trafilatura.readthedocs.io/evaluation.html
evaluation readmehttps://github.com/adbar/trafilatura/blob/master/tests/README.rst
#https://trafilatura.readthedocs.io/#quick-start
installation documentationhttps://trafilatura.readthedocs.io/installation.html
usage documentationhttps://trafilatura.readthedocs.io/usage.html
tutorialshttps://trafilatura.readthedocs.io/tutorials.html
video tutorials playlisthttps://www.youtube.com/watch?v=8GkiOM17t0Q&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci
#https://trafilatura.readthedocs.io/#license
Apache 2.0 licensehttps://www.apache.org/licenses/LICENSE-2.0.html
#https://trafilatura.readthedocs.io/#support
on GitHubhttps://github.com/sponsors/adbar
ko-fi.comhttps://ko-fi.com/adbarbaresi
Contributing pagehttps://github.com/adbar/trafilatura/blob/master/CONTRIBUTING.md
#https://trafilatura.readthedocs.io/#context
wire drawinghttps://en.wikipedia.org/wiki/Wire_drawing
#https://trafilatura.readthedocs.io/#author
contact pagehttps://adrien.barbaresi.eu/
Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extractionhttps://aclanthology.org/2021.acl-demo.15/
Generic Web Content Extraction with Open-Source Softwarehttps://hal.archives-ouvertes.fr/hal-02447264/document
Efficient construction of metadata-enhanced web corporahttps://hal.archives-ouvertes.fr/hal-01371704v2/document
10th Web as Corpus Workshop (WAC-X)https://www.sigwac.org.uk/wiki/WAC-X
#https://trafilatura.readthedocs.io/#citing-trafilatura
https://aclanthology.org/2021.acl-demo.15/
https://doi.org/10.5281/zenodo.3460969
#https://trafilatura.readthedocs.io/#software-ecosystem
https://trafilatura.readthedocs.io/_images/software-ecosystem.png
Bits of Languagehttps://adrien.barbaresi.eu/blog/tag/trafilatura.html
#https://trafilatura.readthedocs.io/#further-documentation
Installationhttps://trafilatura.readthedocs.io/installation.html
Pythonhttps://trafilatura.readthedocs.io/installation.html#python
Trafilatura packagehttps://trafilatura.readthedocs.io/installation.html#trafilatura-package
Additional functionalityhttps://trafilatura.readthedocs.io/installation.html#additional-functionality
Quickstarthttps://trafilatura.readthedocs.io/quickstart.html
With Pythonhttps://trafilatura.readthedocs.io/quickstart.html#with-python
On the command-linehttps://trafilatura.readthedocs.io/quickstart.html#on-the-command-line
Further stepshttps://trafilatura.readthedocs.io/quickstart.html#further-steps
Usagehttps://trafilatura.readthedocs.io/usage.html
Quickstarthttps://trafilatura.readthedocs.io/quickstart.html
Python usagehttps://trafilatura.readthedocs.io/usage-python.html
Command-line usagehttps://trafilatura.readthedocs.io/usage-cli.html
Graphical user interfacehttps://trafilatura.readthedocs.io/usage-gui.html
Usage with Rhttps://trafilatura.readthedocs.io/usage-r.html
APIhttps://trafilatura.readthedocs.io/usage-api.html
Download web pageshttps://trafilatura.readthedocs.io/downloads.html
Web crawlinghttps://trafilatura.readthedocs.io/crawls.html
Deduplicationhttps://trafilatura.readthedocs.io/deduplication.html
URL managementhttps://trafilatura.readthedocs.io/url-management.html
Tutorialshttps://trafilatura.readthedocs.io/tutorials.html
Tutorial: Building a web corpushttps://trafilatura.readthedocs.io/tutorial-corpus.html
Tutorial: Word frequency lists from web pageshttps://trafilatura.readthedocs.io/tutorial-frequencies.html
Tutorial: Validation of TEI fileshttps://trafilatura.readthedocs.io/tutorial-tei.html
Tutorial: DWDS-Korpusdaten reproduzierenhttps://trafilatura.readthedocs.io/tutorial-dwds.html
Tutorial: Text embedding with ChromaDBhttps://trafilatura.readthedocs.io/tutorial-chromadb.html
Tutorial: Retrieval-augmented generation (RAG)https://trafilatura.readthedocs.io/tutorial-rag.html
Tutorial: Building a training corpus with Hugging Face Datasetshttps://trafilatura.readthedocs.io/tutorial-datasets.html
Blog postshttps://trafilatura.readthedocs.io/tutorials.html#blog-posts
Videoshttps://trafilatura.readthedocs.io/tutorials.html#videos
External resourceshttps://trafilatura.readthedocs.io/tutorials.html#external-resources
FAQhttps://trafilatura.readthedocs.io/faq.html
How to extract text from a webpage in Pythonhttps://trafilatura.readthedocs.io/faq.html#how-to-extract-text-from-a-webpage-in-python
Which extraction function should I use?https://trafilatura.readthedocs.io/faq.html#which-extraction-function-should-i-use
How does Trafilatura compare to BeautifulSoup or Scrapy?https://trafilatura.readthedocs.io/faq.html#how-does-trafilatura-compare-to-beautifulsoup-or-scrapy
How to extract from local HTML fileshttps://trafilatura.readthedocs.io/faq.html#how-to-extract-from-local-html-files
How to process many pages efficientlyhttps://trafilatura.readthedocs.io/faq.html#how-to-process-many-pages-efficiently
How to customize what gets extractedhttps://trafilatura.readthedocs.io/faq.html#how-to-customize-what-gets-extracted
How to use results with pandas or other toolshttps://trafilatura.readthedocs.io/faq.html#how-to-use-results-with-pandas-or-other-tools
How to handle large or slow pageshttps://trafilatura.readthedocs.io/faq.html#how-to-handle-large-or-slow-pages
What are common output format gotchas?https://trafilatura.readthedocs.io/faq.html#what-are-common-output-format-gotchas
Troubleshootinghttps://trafilatura.readthedocs.io/troubleshooting.html
Debugginghttps://trafilatura.readthedocs.io/troubleshooting.html#debugging
Extraction problemshttps://trafilatura.readthedocs.io/troubleshooting.html#extraction-problems
Download problemshttps://trafilatura.readthedocs.io/troubleshooting.html#download-problems
Performance problemshttps://trafilatura.readthedocs.io/troubleshooting.html#performance-problems
How extraction workshttps://trafilatura.readthedocs.io/extraction-overview.html
Pipeline at a glancehttps://trafilatura.readthedocs.io/extraction-overview.html#pipeline-at-a-glance
Stageshttps://trafilatura.readthedocs.io/extraction-overview.html#stages
Extraction modeshttps://trafilatura.readthedocs.io/extraction-overview.html#extraction-modes
Metadatahttps://trafilatura.readthedocs.io/extraction-overview.html#metadata
Output formattinghttps://trafilatura.readthedocs.io/extraction-overview.html#output-formatting
Benchmarks and evaluationhttps://trafilatura.readthedocs.io/evaluation.html
External evaluationshttps://trafilatura.readthedocs.io/evaluation.html#external-evaluations
Alternativeshttps://trafilatura.readthedocs.io/evaluation.html#alternatives
Descriptionhttps://trafilatura.readthedocs.io/evaluation.html#description
Results (2026-10-02)https://trafilatura.readthedocs.io/evaluation.html#results-2026-10-02
Older resultshttps://trafilatura.readthedocs.io/evaluation.html#older-results
Core functionshttps://trafilatura.readthedocs.io/corefunctions.html
Extractionhttps://trafilatura.readthedocs.io/corefunctions.html#extraction
Link discoveryhttps://trafilatura.readthedocs.io/corefunctions.html#link-discovery
Helpershttps://trafilatura.readthedocs.io/corefunctions.html#helpers
XML processinghttps://trafilatura.readthedocs.io/corefunctions.html#xml-processing
Settings and customizationhttps://trafilatura.readthedocs.io/settings.html
Configuration filehttps://trafilatura.readthedocs.io/settings.html#configuration-file
Package settingshttps://trafilatura.readthedocs.io/settings.html#package-settings
Deprecations and migrationhttps://trafilatura.readthedocs.io/deprecations.html
Python version supporthttps://trafilatura.readthedocs.io/deprecations.html#python-version-support
Migrating to v2https://trafilatura.readthedocs.io/deprecations.html#migrating-to-v2
Uses & citationshttps://trafilatura.readthedocs.io/used-by.html
Notable projects using this softwarehttps://trafilatura.readthedocs.io/used-by.html#notable-projects-using-this-software
Citations in papershttps://trafilatura.readthedocs.io/used-by.html#citations-in-papers
Publications citing Trafilaturahttps://trafilatura.readthedocs.io/used-by.html#publications-citing-trafilatura
Publications citing Htmldatehttps://trafilatura.readthedocs.io/used-by.html#publications-citing-htmldate
changeloghttps://github.com/adbar/trafilatura/blob/master/HISTORY.md
Running the testshttps://trafilatura.readthedocs.io/tests.html
Installationhttps://trafilatura.readthedocs.io/tests.html#installation
Running the test suitehttps://trafilatura.readthedocs.io/tests.html#running-the-test-suite
Type checking and code stylehttps://trafilatura.readthedocs.io/tests.html#type-checking-and-code-style
Benchmark and evaluationhttps://trafilatura.readthedocs.io/tests.html#benchmark-and-evaluation
Building the documentationhttps://trafilatura.readthedocs.io/tests.html#building-the-documentation
Backgroundhttps://trafilatura.readthedocs.io/background.html
Compendium: Web texts in linguistics and humanitieshttps://trafilatura.readthedocs.io/compendium.html
Finding sources for web corporahttps://trafilatura.readthedocs.io/sources.html
Working with corpus datahttps://trafilatura.readthedocs.io/corpus-data.html
Indexhttps://trafilatura.readthedocs.io/genindex.html
next Installation https://trafilatura.readthedocs.io/installation.html
Trafilatura: discover web content, extract text and metadatahttps://trafilatura.readthedocs.io/
Descriptionhttps://trafilatura.readthedocs.io/#description
Featureshttps://trafilatura.readthedocs.io/#features
Evaluationhttps://trafilatura.readthedocs.io/#evaluation
Quick starthttps://trafilatura.readthedocs.io/#quick-start
Licensehttps://trafilatura.readthedocs.io/#license
Supporthttps://trafilatura.readthedocs.io/#support
Contexthttps://trafilatura.readthedocs.io/#context
Authorhttps://trafilatura.readthedocs.io/#author
Citing Trafilaturahttps://trafilatura.readthedocs.io/#citing-trafilatura
Software ecosystemhttps://trafilatura.readthedocs.io/#software-ecosystem
Further documentationhttps://trafilatura.readthedocs.io/#further-documentation
Show Source https://trafilatura.readthedocs.io/_sources/index.rst.txt
PyData Sphinx Themehttps://pydata-sphinx-theme.readthedocs.io/en/stable/index.html

Viewport: width=device-width, initial-scale=1

URLs of crawlers that visited me.