René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Integrate KiwixStorage","articleBody":"Ours scrapers all depend on S3 and use [`KiwixStorage`](https://github.com/openzim/python-storagelib) for it. That wrapper and repo are mostly untouched (yet working) and would great benefit from being integrated into scraperlib: tests, usage, dependencies update, better API, etc.\n\nNow that we had experience using it, it probably make sense to reduce its API to what's actually used.\n\nIt's true that some non-scraper projects are using it (cardshop and zimfarm to upload to S3), but those are also using scraperlib and we've discussed making scraperlib's dependencies more modular via extras anyway.","author":{"url":"https://github.com/rgaudin","@type":"Person","name":"rgaudin"},"datePublished":"2024-11-26T09:50:07.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/223/python-scraperlib/issues/223"}