René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"spanner: occasional \"google.api_core.exceptions.Aborted: 409 Transaction not found\" error with PingingPool","articleBody":"Given spanner_v1 **VERSION1.11.0**, I am obtaining a transaction from a PingingPool as per\r\n```python\r\n    # Create a session pool that'll periodically refresh every 3 minutes (arbitrary choice value).\r\n    pool = spanner.PingingPool(size=10, default_timeout=5, ping_interval=180)\r\n    background_thread = threading.Thread(target=pool.ping, name='ping-pool')\r\n    background_thread.daemon = True\r\n    background_thread.start()\r\n\r\n    db = client_instance.database(database, pool=pool)\r\n    if not db.exists():\r\n        raise ProgrammingError(\"database '%s' does not exist.\" % database)\r\n\r\n    sess = db.session()\r\n    ...\r\n    # Then later obtaining a transaction and holding it for a long-ish time\r\n    txn = sess.transaction()\r\n    txn.begin()\r\n    # Do a bunch of operations with the operation\r\n    ...\r\n    txn.commit()\r\n```\r\nand I can confirm that pool isn't being used concurrently, but I've seen a test failure with\r\n```shell\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py\", line 726, in _end_unary_response_blocking\r\n    raise _InactiveRpcError(state)\r\ngrpc._channel._InactiveRpcError: \u003c_InactiveRpcError of RPC that terminated with:\r\n\tstatus = StatusCode.ABORTED\r\n\tdetails = \"Transaction not found\"\r\n\tdebug_error_string = \"{\"created\":\"@1580854844.873538358\",\"description\":\"Error received from peer ipv4:172.217.204.95:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Transaction not found\",\"grpc_status\":10}\"\r\n```\r\nand in full detail\r\n\r\n\u003cdetails\u003e\r\n\r\n```shell\r\nTraceback (most recent call last):\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 57, in error_remapped_callable\r\n    return callable_(*args, **kwargs)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py\", line 824, in __call__\r\n    return _end_unary_response_blocking(state, call, False, None)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py\", line 726, in _end_unary_response_blocking\r\n    raise _InactiveRpcError(state)\r\ngrpc._channel._InactiveRpcError: \u003c_InactiveRpcError of RPC that terminated with:\r\n\tstatus = StatusCode.ABORTED\r\n\tdetails = \"Transaction not found\"\r\n\tdebug_error_string = \"{\"created\":\"@1580854844.873538358\",\"description\":\"Error received from peer ipv4:172.217.204.95:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Transaction not found\",\"grpc_status\":10}\"\r\n\u003e\r\nThe above exception was the direct cause of the following exception:\r\nTraceback (most recent call last):\r\n  File \"runtests.py\", line 507, in \u003cmodule\u003e\r\n    options.exclude_tags,\r\n  File \"runtests.py\", line 294, in django_tests\r\n    extra_tests=extra_tests,\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/runner.py\", line 629, in run_tests\r\n    old_config = self.setup_databases(aliases=databases)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/runner.py\", line 554, in setup_databases\r\n    self.parallel, **kwargs\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/utils.py\", line 174, in setup_databases\r\n    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/django/creation.py\", line 33, in create_test_db\r\n    super().create_test_db(*args, **kwargs)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/base/creation.py\", line 72, in create_test_db\r\n    run_syncdb=True,\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/core/management/__init__.py\", line 148, in call_command\r\n    return command.execute(*args, **defaults)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/core/management/base.py\", line 364, in execute\r\n    output = self.handle(*args, **options)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/core/management/base.py\", line 83, in wrapped\r\n    res = handle_func(*args, **kwargs)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/core/management/commands/migrate.py\", line 257, in handle\r\n    self.verbosity, self.interactive, connection.alias, apps=post_migrate_apps, plan=plan,\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/core/management/sql.py\", line 51, in emit_post_migrate_signal\r\n    **kwargs\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/dispatch/dispatcher.py\", line 175, in send\r\n    for receiver in self._live_receivers(sender)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/dispatch/dispatcher.py\", line 175, in \u003clistcomp\u003e\r\n    for receiver in self._live_receivers(sender)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/management/__init__.py\", line 83, in create_permissions\r\n    Permission.objects.using(using).bulk_create(perms)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py\", line 468, in bulk_create\r\n    self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py\", line 1211, in _batched_insert\r\n    self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py\", line 1186, in _insert\r\n    return query.get_compiler(using=using).execute_sql(return_id)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py\", line 1368, in execute_sql\r\n    cursor.execute(sql, params)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py\", line 67, in execute\r\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\r\n    return executor(sql, params, many, context)\r\n  File \"/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py\", line 84, in _execute\r\n    return self.cursor.execute(sql, params)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py\", line 87, in execute\r\n    self.__handle_insert(self.__get_txn(), sql, args or None)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py\", line 128, in __handle_insert\r\n    res = txn.execute_update(sql, params=params, param_types=param_types)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/transaction.py\", line 202, in execute_update\r\n    metadata=metadata,\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py\", line 810, in execute_sql\r\n    request, retry=retry, timeout=timeout, metadata=metadata\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py\", line 143, in __call__\r\n    return wrapped_func(*args, **kwargs)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py\", line 286, in retry_wrapped_func\r\n    on_error=on_error,\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py\", line 184, in retry_target\r\n    return target()\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/timeout.py\", line 214, in func_with_timeout\r\n    return func(*args, **kwargs)\r\n  File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 59, in error_remapped_callable\r\n    six.raise_from(exceptions.from_grpc_error(exc), exc)\r\n  File \"\u003cstring\u003e\", line 3, in raise_from\r\ngoogle.api_core.exceptions.Aborted: 409 Transaction not found\r\n```\r\n\u003c/details\u003e","author":{"url":"https://github.com/odeke-em","@type":"Person","name":"odeke-em"},"datePublished":"2020-02-04T22:32:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/10/python-spanner/issues/10"}