René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"additional fix for 504 errors","articleBody":"**Describe the bug**\r\n@justinpolygon I found another place in the code that needs to be updated in order for the new #525 retry logic to work properly. The following line (a remnant of the old code) needs to be removed:\r\n\r\nhttps://github.com/polygon-io/client-python/blob/master/polygon/rest/base.py#L113\r\n\r\nThis line effectively overrides the new retry_strategy object specified in the urllib3.PoolManager initialization line, so it prevents that logic from taking effect. I discovered this when I backed out my tenacity workaround and got a 504 error last night.\r\n\r\n**To Reproduce**\r\nI found a way to reproduce this problem by monkey-patching the `_make_request` method in urllib3 to randomly return 504 instead of the actual error code, which forces an exercise of the retry logic. To implement this test, make the following change to polygon/rest/base.py. Right after the import urllib3 line, insert the following code:\r\n\r\n```python\r\noriginal_make_request = urllib3.connectionpool.HTTPConnectionPool._make_request\r\ndef monkey_patched_make_request(self, conn, method, url, **kwargs):\r\n    response = original_make_request(self, conn, method, url, **kwargs)\r\n    import random\r\n    if random.random() \u003c 0.90: response.status = 504\r\n    return response\r\nurllib3.connectionpool.HTTPConnectionPool._make_request = monkey_patched_make_request\r\n```\r\n\r\nHere is a sample script to test the library with the above change:\r\n\r\n```python\r\nimport polygon\r\nfrom pprint import pprint\r\n\r\npolygon_api = polygon.RESTClient(POLYGON_KEY_REDACTED, retries=2, trace=True, verbose=True)\r\n\r\ntrades = [*polygon_api.list_trades('AAPL', timestamp_gt=1695944980000000000, timestamp_lte=1695945000000000000)]\r\npprint(trades)\r\n```\r\n\r\nWith the current code, running the sample script results in the following kind of intermittent error, which indicates that retries aren't actually happening:\r\n\r\n```\r\npolygon.exceptions.BadResponse: {\"results\"': [...]}\r\n```\r\n\r\nWhen I delete line 113 from polygon/rest/base.api, I get a totally different intermittent error:\r\n\r\n```\r\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.polygon.io', port=443): Max retries exceeded with url: /v3/trades/AAPL?timestamp.lte=1695945000000000000\u0026timestamp.gt=1695944980000000000 (Caused by ResponseError('too many 504 error responses'))\r\n```\r\nThis indicates that the retries are actually happening.","author":{"url":"https://github.com/kartiksubbarao","@type":"Person","name":"kartiksubbarao"},"datePublished":"2023-09-29T01:48:47.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/528/client-python/issues/528"}