René's URL Explorer Experiment


Title: Command-line usage — Trafilatura 2.3.1 documentation

Open Graph Title: Command-line usage

Description: Trafilatura offers a robust CLI. Learn how to download and extract text from HTML web pages without writing code, including parallel processing and data mining capabilities.

Open Graph Description: Introduction: Trafilatura offers a robust command-line interface and can be conveniently used without writing code. For the very first steps: Multilingual Introduction to the command-line interface...

Opengraph URL: https://trafilatura.readthedocs.io/usage-cli.html

direct link

Domain: trafilatura.readthedocs.io

og:typewebsite
og:site_nameTrafilatura
og:imagehttps://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-logo.png
og:image:altTrafilatura
docsearch:languageen
docsearch:version2.3.1
readthedocs-project-slugtrafilatura
readthedocs-version-sluglatest
readthedocs-resolver-filename/usage-cli.html
readthedocs-http-status200

Links:

Skip to main contenthttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#main-content
https://trafilatura.readthedocs.io/en/latest/index.html
Installation https://trafilatura.readthedocs.io/en/latest/installation.html
Quickstart https://trafilatura.readthedocs.io/en/latest/quickstart.html
Usage https://trafilatura.readthedocs.io/en/latest/usage.html
Tutorials https://trafilatura.readthedocs.io/en/latest/tutorials.html
FAQ https://trafilatura.readthedocs.io/en/latest/faq.html
Troubleshooting https://trafilatura.readthedocs.io/en/latest/troubleshooting.html
How extraction works https://trafilatura.readthedocs.io/en/latest/extraction-overview.html
Benchmarks and evaluation https://trafilatura.readthedocs.io/en/latest/evaluation.html
Core functions https://trafilatura.readthedocs.io/en/latest/corefunctions.html
Settings and customization https://trafilatura.readthedocs.io/en/latest/settings.html
Deprecations and migration https://trafilatura.readthedocs.io/en/latest/deprecations.html
Uses & citations https://trafilatura.readthedocs.io/en/latest/used-by.html
Running the tests https://trafilatura.readthedocs.io/en/latest/tests.html
Background https://trafilatura.readthedocs.io/en/latest/background.html
Blog https://adrien.barbaresi.eu/blog/tag/trafilatura.html
GitHubhttps://github.com/adbar/trafilatura
Installation https://trafilatura.readthedocs.io/en/latest/installation.html
Quickstart https://trafilatura.readthedocs.io/en/latest/quickstart.html
Usage https://trafilatura.readthedocs.io/en/latest/usage.html
Tutorials https://trafilatura.readthedocs.io/en/latest/tutorials.html
FAQ https://trafilatura.readthedocs.io/en/latest/faq.html
Troubleshooting https://trafilatura.readthedocs.io/en/latest/troubleshooting.html
How extraction works https://trafilatura.readthedocs.io/en/latest/extraction-overview.html
Benchmarks and evaluation https://trafilatura.readthedocs.io/en/latest/evaluation.html
Core functions https://trafilatura.readthedocs.io/en/latest/corefunctions.html
Settings and customization https://trafilatura.readthedocs.io/en/latest/settings.html
Deprecations and migration https://trafilatura.readthedocs.io/en/latest/deprecations.html
Uses & citations https://trafilatura.readthedocs.io/en/latest/used-by.html
Running the tests https://trafilatura.readthedocs.io/en/latest/tests.html
Background https://trafilatura.readthedocs.io/en/latest/background.html
Blog https://adrien.barbaresi.eu/blog/tag/trafilatura.html
GitHubhttps://github.com/adbar/trafilatura
Quickstarthttps://trafilatura.readthedocs.io/en/latest/quickstart.html
Python usagehttps://trafilatura.readthedocs.io/en/latest/usage-python.html
Command-line usagehttps://trafilatura.readthedocs.io/en/latest/usage-cli.html
Graphical user interfacehttps://trafilatura.readthedocs.io/en/latest/usage-gui.html
Usage with Rhttps://trafilatura.readthedocs.io/en/latest/usage-r.html
APIhttps://trafilatura.readthedocs.io/en/latest/usage-api.html
Download web pageshttps://trafilatura.readthedocs.io/en/latest/downloads.html
Web crawlinghttps://trafilatura.readthedocs.io/en/latest/crawls.html
Deduplicationhttps://trafilatura.readthedocs.io/en/latest/deduplication.html
URL managementhttps://trafilatura.readthedocs.io/en/latest/url-management.html
https://trafilatura.readthedocs.io/en/latest/index.html
Usagehttps://trafilatura.readthedocs.io/en/latest/usage.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#command-line-usage
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#introduction
Introduction to the command-line interfacehttps://tutorial.djangogirls.org/en/intro_to_command_line/
Introduction to Cultural Analytics & Pythonhttps://melaniewalsh.github.io/Intro-Cultural-Analytics/01-Command-Line/01-The-Command-Line.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#quickstart
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#extraction-parameters
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#choice-of-html-elements
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#output-format
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#fast-mode
how extraction workshttps://trafilatura.readthedocs.io/en/latest/extraction-overview.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#optimizing-for-precision-and-recall
troubleshootinghttps://trafilatura.readthedocs.io/en/latest/troubleshooting.html
how extraction workshttps://trafilatura.readthedocs.io/en/latest/extraction-overview.html#extraction-modes
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#language-identification
installedhttps://trafilatura.readthedocs.io/en/latest/installation.html
py3langid packagehttps://github.com/adbar/py3langid
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#metadata
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#deduplication
deduplicationhttps://trafilatura.readthedocs.io/en/latest/deduplication.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#blacklist
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#changing-default-settings
documentation page on settingshttps://trafilatura.readthedocs.io/en/latest/settings.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#process-files-locally
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#process-a-list-of-links
user-agenthttps://en.wikipedia.org/wiki/User_agent
page on downloadshttps://trafilatura.readthedocs.io/en/latest/downloads.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#source-based-file-names
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#internet-archive
Internet Archivehttps://web.archive.org/
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#link-discovery
web feedshttps://en.wikipedia.org/wiki/Web_feed
sitemapshttps://en.wikipedia.org/wiki/Sitemaps
tutorial on content discoveryhttps://trafilatura.readthedocs.io/en/latest/tutorial-corpus.html#content-discovery
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#feeds
Extracting links from web feedshttps://www.youtube.com/watch?v=NW2ISdOx08M&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci&index=2&t=398s
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#id4
Using sitemaps to crawl websiteshttps://adrien.barbaresi.eu/blog/using-sitemaps-crawl-websites.html
Listing all website contents with sitemapshttps://www.youtube.com/watch?v=uWUyhxciTOs&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci&index=3&t=330s
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#web-crawling
page on web crawlinghttps://trafilatura.readthedocs.io/en/latest/crawls.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#explore-and-probe
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#url-inspection-prior-to-download-and-processing
Using sitemaps to crawl websiteshttps://adrien.barbaresi.eu/blog/using-sitemaps-crawl-websites.html
tutorial on link filteringhttps://trafilatura.readthedocs.io/en/latest/tutorial-corpus.html#link-filtering
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#deprecations
deprecations and migrationhttps://trafilatura.readthedocs.io/en/latest/deprecations.html
#https://trafilatura.readthedocs.io/en/latest/usage-cli.html#further-information
how to modify the default settingshttps://trafilatura.readthedocs.io/en/latest/settings.html
Settings and customizationhttps://trafilatura.readthedocs.io/en/latest/settings.html
Python usagehttps://trafilatura.readthedocs.io/en/latest/usage-python.html
previous Python usage https://trafilatura.readthedocs.io/en/latest/usage-python.html
next Graphical user interface https://trafilatura.readthedocs.io/en/latest/usage-gui.html
Introductionhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#introduction
Quickstarthttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#quickstart
Extraction parametershttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#extraction-parameters
Choice of HTML elementshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#choice-of-html-elements
Output formathttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#output-format
Fast modehttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#fast-mode
Optimizing for precision and recallhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#optimizing-for-precision-and-recall
Language identificationhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#language-identification
Metadatahttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#metadata
Deduplicationhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#deduplication
Blacklisthttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#blacklist
Changing default settingshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#changing-default-settings
Process files locallyhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#process-files-locally
Process a list of linkshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#process-a-list-of-links
Source-based file nameshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#source-based-file-names
Internet Archivehttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#internet-archive
Link discoveryhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#link-discovery
Feedshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#feeds
Sitemapshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#id4
Web crawlinghttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#web-crawling
Explore and probehttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#explore-and-probe
URL inspection prior to download and processinghttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#url-inspection-prior-to-download-and-processing
Deprecationshttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#deprecations
Further informationhttps://trafilatura.readthedocs.io/en/latest/usage-cli.html#further-information
Show Source https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt
PyData Sphinx Themehttps://pydata-sphinx-theme.readthedocs.io/en/stable/index.html

Viewport: width=device-width, initial-scale=1

URLs of crawlers that visited me.