adbar / trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

https://trafilatura.readthedocs.io

adbar/trafilatura Issues

Loss format or data when li contains p
Updated 11 days ago
Duplicated lines when nested in <article> and <main>, with <br> in front
Updated 13 days ago1
Documentation: on precision
Updated 17 days ago1
Extracting full text from an URL returns None
Closed 22 days ago2
CLI: better control of output file names
Updated 22 days ago2
AttributeError in prune_unwanted_nodes
Closed 22 days ago2
Backticks produce extra line breaks
Updated 25 days ago1
Support for sidemap parsing from text instead of urls
Updated a month ago1
Performance bottleneck in `prune_unwanted_nodes` causing 200ms per call
Updated a month ago1
Explicitly and fully support type hinting
Closed a month ago
Review input type for `is_probably_readerable()` function
Updated a month ago
Documentation about settings could use examples
Updated a month ago1
Download multiple urls with download timeout
Closed 2 months ago2
Extraction: move `max_tree_size` to config file
Closed 2 months ago
setup: set `__all__` in `__init__.py`
Closed 2 months ago
Add magic_html to benchmarks
Closed 2 months ago2
Crawler doesn't extract any links from Google Cloud documentation website
Closed 4 months ago6
Downloads: fully use information from both `config` and `options` variables
Closed 2 months ago
CLI downloads: make sure all user-specified options are used
Closed 2 months ago4
`bare_extraction()`: deprecate `as_dict` parameter
Closed 2 months ago
`extract()`: replace `no_fallback` argument by `fast`
Closed 2 months ago
feat(cli/lib): Add tqdm based progress bar as an option
Updated 2 months ago1
I can't extract main content from this html,could anyone help me?
Closed 2 months ago1
extract function runs indefinitely on large HTML body content
Closed 2 months ago1
Focused crawler returns 404 response for robots.txt and stops crawling
Closed 2 months ago1
Deprecate `fetch_url(decode=False)`
Closed 2 months ago
HTML_TAG_MAPPING error during scrape
Closed 2 months ago2
Review HTML element list and conversion
Updated 2 months ago
setup: use `pyproject.toml` file
Closed 3 months ago
Remove deprecations (mostly CLI)
Closed 3 months ago
Trafilatura crashing due to `options` variable not backfilled yet
Closed 3 months ago1
Javascript Version has landed. 🚀
Closed 3 months ago3
Empty Results When Using Spider Function with Category URL
Updated 3 months ago5
Link on the quickstart page to the overview notebook is broken
Closed 4 months ago1
Docs: add page explaining how to run tests
Updated 4 months ago
Downloads: add support to switch between proxies
Updated 4 months ago
ImportError: lxml.html.clean module is now a separate project
Closed 4 months ago2
ValueError in xml
Closed 4 months ago3
How can I set the proxy IP port and userAgent to avoid the web anti-crawler mechanism?
Closed 5 months ago2
spider: restrict search to given URL pattern
Closed 4 months ago
trafilatura version > 1.10.0 doesnt fetch images
Closed 5 months ago3
Investigate spacing in element tails
Updated 5 months ago3
AttributeError in prune_unwanted_sections
Closed 5 months ago3
Bug or feature, I'm not sure!
Closed 5 months ago1
Faulty extraction for very short documents
Updated 5 months ago4
Duplicating sections, removing spaces between words, simple example
Closed 5 months ago
MemoryError in table conversion
Closed 5 months ago2
Extraction with `include_images=True` takes too much time
Closed 5 months ago3
CLI: Trigger formatting parameter when the output is in Markdown format
Closed 5 months ago
Validate value of `output_format` in `extract()` and `bare_extraction()`
Closed 5 months ago