René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"On a multi-thread context, the use of this method could lead to \"HTTPSConnectionPool ... Max retries exceeded with url...\"","articleBody":"https://github.com/clearbit/clearbit-python/blob/5594e824663b8aaa615e1ec15af61d309239716a/clearbit/resource.py#L46\r\n\r\nThe use of \"requests\" component library on a multi-thread context with a large number of concurrent threads requests data to Clearbit Servers, as an example, using the Reveal API to find company data by IP. We could got an error from server. But, this is not an issue on server side, because server has a limit and this is fine. The problem is how to handle this issue on client side.\r\n\r\nSo, I made an improve on my own using the code below:\r\n\r\n_A little brief on this code: Find company by IP._\r\n\r\nTo highlight the improvement, the code add a Retry strategy to handle issues when the server returns the status code [429, 500, 502, 503, 504. Just remembering that this is from my case, we should refactor this using a factory + configuration design model to avoid hardocoded configuration on this end.\r\n\r\n```python\r\n\r\n@staticmethod\r\ndef __createSession__():\r\n    session = requests.Session()\r\n    retry = Retry(\r\n        total=3,\r\n        backoff_factor=1,\r\n        status_forcelist=[429, 500, 502, 503, 504],\r\n        method_whitelist=[\"HEAD\", \"GET\", \"OPTIONS\"]\r\n    )\r\n    adapter = HTTPAdapter(max_retries=retry)\r\n    session.mount('http://', adapter)\r\n    session.mount('https://', adapter)\r\n    return session\r\n\r\n\r\n def findCompanyInfoByIp(self, ip: string) -\u003e ClearbitResult:\r\n        if ip is None or ip == \"\":\r\n            return ErrorClearbitResult(f\"Can't find clearbit data with a valid ip\")\r\n        url = f\"{BASE_URL}/find?ip={urllib.parse.quote(ip)}\"\r\n        try:\r\n            response = \\\r\n                self.__requestSession.get(\r\n                    url,\r\n                    auth=BearerAuth(self.__authToken))\r\n            return ClearbitResult(response)\r\n        except Exception as e:\r\n            errorMessage = f\"Error requesting Clearbit at [{url}]: {e}\"\r\n            self.__logger.error(errorMessage)\r\n            return ErrorClearbitResult(errorMessage)\r\n```\r\n\r\nThis solution helps me to ensure 100 or maybe more threands to requests Clearbit Reveal API using a simple retry strategy.\r\n\r\nCould this be an improvement to help this library?","author":{"url":"https://github.com/fabiofalavinha","@type":"Person","name":"fabiofalavinha"},"datePublished":"2021-09-13T12:51:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/35/clearbit-python/issues/35"}