René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"'test_cluster_create' systest flakes with 'TimeoutError'","articleBody":"From [this failed systest build](https://source.cloud.google.com/results/invocations/924fbb8c-f387-482d-8767-c16abd2de9e2/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-bigtable%2Fpresubmit%2Fsystem-3.8/log):\r\n\r\n```python\r\n_____________________________ test_cluster_create ______________________________\r\n\r\ntarget = functools.partial(\u003cbound method PollingFuture._done_or_raise of \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\u003e)\r\npredicate = \u003cfunction if_exception_type.\u003clocals\u003e.if_exception_type_predicate at 0x7f4d5ef43ee0\u003e\r\nsleep_generator = \u003cgenerator object exponential_sleep_generator at 0x7f4d5c1b1660\u003e\r\ndeadline = 30, on_error = None\r\n\r\n    def retry_target(target, predicate, sleep_generator, deadline, on_error=None):\r\n        \"\"\"Call a function and retry if it fails.\r\n\r\n        This is the lowest-level retry helper. Generally, you'll use the\r\n        higher-level retry helper :class:`Retry`.\r\n\r\n        Args:\r\n            target(Callable): The function to call and retry. This must be a\r\n                nullary function - apply arguments with `functools.partial`.\r\n            predicate (Callable[Exception]): A callable used to determine if an\r\n                exception raised by the target should be considered retryable.\r\n                It should return True to retry or False otherwise.\r\n            sleep_generator (Iterable[float]): An infinite iterator that determines\r\n                how long to sleep between retries.\r\n            deadline (float): How long to keep retrying the target. The last sleep\r\n                period is shortened as necessary, so that the last retry runs at\r\n                ``deadline`` (and not considerably beyond it).\r\n            on_error (Callable[Exception]): A function to call while processing a\r\n                retryable exception.  Any error raised by this function will *not*\r\n                be caught.\r\n\r\n        Returns:\r\n            Any: the return value of the target function.\r\n\r\n        Raises:\r\n            google.api_core.RetryError: If the deadline is exceeded while retrying.\r\n            ValueError: If the sleep generator stops yielding values.\r\n            Exception: If the target raises a method that isn't retryable.\r\n        \"\"\"\r\n        if deadline is not None:\r\n            deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(\r\n                seconds=deadline\r\n            )\r\n        else:\r\n            deadline_datetime = None\r\n\r\n        last_exc = None\r\n\r\n        for sleep in sleep_generator:\r\n            try:\r\n\u003e               return target()\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:190:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\r\nretry = \u003cgoogle.api_core.retry.Retry object at 0x7f4d5f0bad60\u003e\r\n\r\n    def _done_or_raise(self, retry=DEFAULT_RETRY):\r\n        \"\"\"Check if the future is done and raise if it's not.\"\"\"\r\n        kwargs = {} if retry is DEFAULT_RETRY else {\"retry\": retry}\r\n\r\n        if not self.done(**kwargs):\r\n\u003e           raise _OperationNotComplete()\r\nE           google.api_core.future.polling._OperationNotComplete\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:89: _OperationNotComplete\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nself = \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\r\ntimeout = 30, retry = \u003cgoogle.api_core.retry.Retry object at 0x7f4d5f0bad60\u003e\r\n\r\n    def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY):\r\n        \"\"\"Poll and wait for the Future to be resolved.\r\n\r\n        Args:\r\n            timeout (int):\r\n                How long (in seconds) to wait for the operation to complete.\r\n                If None, wait indefinitely.\r\n        \"\"\"\r\n        if self._result_set:\r\n            return\r\n\r\n        retry_ = self._retry.with_deadline(timeout)\r\n\r\n        try:\r\n            kwargs = {} if retry is DEFAULT_RETRY else {\"retry\": retry}\r\n\u003e           retry_(self._done_or_raise)(**kwargs)\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:110:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nargs = (), kwargs = {}\r\ntarget = functools.partial(\u003cbound method PollingFuture._done_or_raise of \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\u003e)\r\nsleep_generator = \u003cgenerator object exponential_sleep_generator at 0x7f4d5c1b1660\u003e\r\n\r\n    @functools.wraps(func)\r\n    def retry_wrapped_func(*args, **kwargs):\r\n        \"\"\"A wrapper that calls target function with retry.\"\"\"\r\n        target = functools.partial(func, *args, **kwargs)\r\n        sleep_generator = exponential_sleep_generator(\r\n            self._initial, self._maximum, multiplier=self._multiplier\r\n        )\r\n\u003e       return retry_target(\r\n            target,\r\n            self._predicate,\r\n            sleep_generator,\r\n            self._deadline,\r\n            on_error=on_error,\r\n        )\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:283:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\ntarget = functools.partial(\u003cbound method PollingFuture._done_or_raise of \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\u003e)\r\npredicate = \u003cfunction if_exception_type.\u003clocals\u003e.if_exception_type_predicate at 0x7f4d5ef43ee0\u003e\r\nsleep_generator = \u003cgenerator object exponential_sleep_generator at 0x7f4d5c1b1660\u003e\r\ndeadline = 30, on_error = None\r\n\r\n    def retry_target(target, predicate, sleep_generator, deadline, on_error=None):\r\n        \"\"\"Call a function and retry if it fails.\r\n\r\n        This is the lowest-level retry helper. Generally, you'll use the\r\n        higher-level retry helper :class:`Retry`.\r\n\r\n        Args:\r\n            target(Callable): The function to call and retry. This must be a\r\n                nullary function - apply arguments with `functools.partial`.\r\n            predicate (Callable[Exception]): A callable used to determine if an\r\n                exception raised by the target should be considered retryable.\r\n                It should return True to retry or False otherwise.\r\n            sleep_generator (Iterable[float]): An infinite iterator that determines\r\n                how long to sleep between retries.\r\n            deadline (float): How long to keep retrying the target. The last sleep\r\n                period is shortened as necessary, so that the last retry runs at\r\n                ``deadline`` (and not considerably beyond it).\r\n            on_error (Callable[Exception]): A function to call while processing a\r\n                retryable exception.  Any error raised by this function will *not*\r\n                be caught.\r\n\r\n        Returns:\r\n            Any: the return value of the target function.\r\n\r\n        Raises:\r\n            google.api_core.RetryError: If the deadline is exceeded while retrying.\r\n            ValueError: If the sleep generator stops yielding values.\r\n            Exception: If the target raises a method that isn't retryable.\r\n        \"\"\"\r\n        if deadline is not None:\r\n            deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(\r\n                seconds=deadline\r\n            )\r\n        else:\r\n            deadline_datetime = None\r\n\r\n        last_exc = None\r\n\r\n        for sleep in sleep_generator:\r\n            try:\r\n                return target()\r\n\r\n            # pylint: disable=broad-except\r\n            # This function explicitly must deal with broad exceptions.\r\n            except Exception as exc:\r\n                if not predicate(exc):\r\n                    raise\r\n                last_exc = exc\r\n                if on_error is not None:\r\n                    on_error(exc)\r\n\r\n            now = datetime_helpers.utcnow()\r\n\r\n            if deadline_datetime is not None:\r\n                if deadline_datetime \u003c= now:\r\n\u003e                   raise exceptions.RetryError(\r\n                        \"Deadline of {:.1f}s exceeded while calling {}\".format(\r\n                            deadline, target\r\n                        ),\r\n                        last_exc,\r\n                    ) from last_exc\r\nE                   google.api_core.exceptions.RetryError: Deadline of 30.0s exceeded while calling functools.partial(\u003cbound method PollingFuture._done_or_raise of \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\u003e), last exception:\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:205: RetryError\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nadmin_instance_populated = \u003cgoogle.cloud.bigtable.instance.Instance object at 0x7f4d5eef10a0\u003e\r\nadmin_instance_id = 'g-c-p-1634667466967', skip_on_emulator = None\r\n\r\n    def test_cluster_create(\r\n        admin_instance_populated, admin_instance_id, skip_on_emulator,\r\n    ):\r\n        alt_cluster_id = f\"{admin_instance_id}-c2\"\r\n        alt_location_id = \"us-central1-f\"\r\n        serve_nodes = 2\r\n\r\n        cluster_2 = admin_instance_populated.cluster(\r\n            alt_cluster_id,\r\n            location_id=alt_location_id,\r\n            serve_nodes=serve_nodes,\r\n            default_storage_type=(enums.StorageType.SSD),\r\n        )\r\n        operation = cluster_2.create()\r\n\u003e       operation.result(timeout=30)  # Ensure the operation completes.\r\n\r\ntests/system/test_instance_admin.py:576:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:132: in result\r\n    self._blocking_poll(timeout=timeout, **kwargs)\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = \u003cgoogle.api_core.operation.Operation object at 0x7f4d5c321d30\u003e\r\ntimeout = 30, retry = \u003cgoogle.api_core.retry.Retry object at 0x7f4d5f0bad60\u003e\r\n\r\n    def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY):\r\n        \"\"\"Poll and wait for the Future to be resolved.\r\n\r\n        Args:\r\n            timeout (int):\r\n                How long (in seconds) to wait for the operation to complete.\r\n                If None, wait indefinitely.\r\n        \"\"\"\r\n        if self._result_set:\r\n            return\r\n\r\n        retry_ = self._retry.with_deadline(timeout)\r\n\r\n        try:\r\n            kwargs = {} if retry is DEFAULT_RETRY else {\"retry\": retry}\r\n            retry_(self._done_or_raise)(**kwargs)\r\n        except exceptions.RetryError:\r\n\u003e           raise concurrent.futures.TimeoutError(\r\n                \"Operation did not complete within the designated \" \"timeout.\"\r\n            )\r\nE           concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout.\r\n\r\n.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:112: TimeoutError\r\n```","author":{"url":"https://github.com/tseaver","@type":"Person","name":"tseaver"},"datePublished":"2021-10-19T18:42:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/451/python-bigtable/issues/451"}