René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Implement datastore v1beta3","articleBody":"Current plan of attack:\n- [x] Remove `--plugin=protoc-gen-grpc=$(GRPC_PLUGIN)` from \"helper\" modules that don't use gRPC (e.g. `google.bigtable.v1.bigtable_data`) (this way sub-packages that don't need gRPC can use them) (#1354)\n- [x] Ditch shared generated modules for those included with [`protobuf`](https://pypi.python.org/pypi/protobuf) and [`googleapis-common-protos`](https://pypi.python.org/pypi/googleapis-common-protos) (#1353) ~~Move shared (i.e. non-Bigtable specific) modules from `gcloud.bigtable._generated` to `gcloud._generated` (this way all sub-packages can have access to shared protobuf message classes) and then subsequently update the `bigtable` imports to accommodate this change~~\n- [x] Pull out [`_get_pb_property_value`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/94c007dea5a2080cd4dfef0c8d62a6c30d82000e/gcloud/bigtable/cluster.py#L50-L71) in `gcloud.bigtable.cluster` into core (`gcloud._helpers`) and separate out a `_has_field` piece since `pb_message.HasField` doesn't work on non-message fields in `proto3` (#1329)\n- [x] Update `Makefile` to reflect changes above (i.e. make sure it can be run without changing the code) (See #1316)\n- [x] Update `Makefile` to automatically add the `_foo.proto` files to the repo (See #1317)\n- [x] Fold `Batch.add_auto_id_entity` into `Batch.put` (the `auto_id_entity` mutation is removed in favor of just using `insert` with a partial key) (see #1296)\n- [x] Loosen `Batch` implementation's reliance on `Mutation` message class (it is used in a `LookupRequest` and their combined structure changes in `v1beta3`) (See #1306, #1319, #1341)\n- [x] Remove use of `serializable` in `Client.transaction` and `Transaction` (it is no longer an option in `v1beta3`, so we will curtail it's use in `v1beta2` before the switch) (See #1294)\n- [x] Manually partition the `_datastore_v1_pb2.py` module into 3 modules that just import portions of the namespace. These modules (`datastore_pb2.py`, `entity_pb2.py`, `query_pb2.py`) will be according to the membership in the new [proto definitions](https://github.com/google/googleapis/tree/12dd2eafc6ba6d011c4079cbfb246b60f9fdc28d/google/datastore/v1beta3) for `v1beta3`. (See #1297, #1299, #1301, #1328)\n- [x] Switch over imports from directly using `_datastore_v1_pb2.py` to using our \"shim\" imports to mock the structure of `v1beta3`. (See #1297, #1299, #1301, #1328)\n- [x] Make sure `API_BASE_URL` for the `Connection` class is used (rather than a parent's version) (See #1293)\n- [x] Change `Client.dataset_id` to `Client.project` (in advance of the rename to `PartitionId.project_id` in `v1beta3`) (#1330)\n- [x] Change the return type of `Connection.commit` to be a tuple of `index_updates` and `mutation_results` (they were previously on the same result object but are being split apart in `v1beta3`) (See #1314)\n- [x] Replace uses of `HasField` for non-message values with `_has_field` (see above) (#1329)\n- [x] Remove `v1beta2` generated `pb2` file and old `.proto` definition (can also remove `_datastore_v1_pb2.py` from the `pylintrc_default` ignored file) and delete our \"shim\" modules (#1355, #1428)\n- [x] Update `Makefile` to incorporate protobuf definitions for `v1beta3` (I did this in a [side-project](https://github.com/dhermes/datastore-v1beta3-migration), #1353, #1354, #1355, #1428)\n- [x] ~~Rewrite our \"shim\" imports to use the actual `datastore._generated` files for `v1beta3`.~~\n- [x] Switch `API_BASE_URL` from `https://www.googleapis.com` to `https://datastore.googleapis.com` and drop `'https://www.googleapis.com/auth/userinfo.email` from the scope list (#1339, #1406)\n- [x] Accommodate renames / retyped / removed\n  - [x] `CommitRequest.mutation  --\u003e  CommitRequest.mutations` (#1461)\n  - [x] `LookupRequest.key  --\u003e  LookupRequest.keys` (~~#1358~~, #1456)\n  - [x] `AllocateIdsResponse.key  --\u003e  AllocateIdsResponse.keys` (~~#1358~~, #1456)\n  - [x] `AllocateIdsRequest.key  --\u003e  AllocateIdsRequest.keys` (~~#1358~~, #1456)\n  - [x] `Key.path_element  --\u003e  Key.path` (~~#1360~~, #1457)\n  - [x] `Entity.property  --\u003e  Entity.properties` (#1458)\n  - [x] `Query.group_by  --\u003e  Query.distinct_on` (~~#1357~~, #1455)\n  - [x] `Query.limit  --\u003e  Query.limit.value` (~~#1357~~, #1455)\n  - [x] `QueryResultBatch.entity_result  --\u003e  QueryResultBatch.entity_results` (~~#1357~~, #1455)\n  - [x] `PartitionId.namespace  --\u003e  PartitionId.namespace_id` (~~#1359~~, #1452)\n  - [x] `PartitionId.dataset_id  --\u003e  PartitionId.project_id` (~~#1359~~, #1452)\n  - [x] `Value.indexed  --\u003e  Value.exclude_from_indexes` (~~#1365~~, #1453)\n  - [x] `Value.list_value  --\u003e  Value.array_value` (#1460)\n  - [x] `Value.null_value` added (#1464)\n  - [x] `Value.timestamp_microseconds_value  --\u003e  Value.timestamp_value` (with type change) (#1361, #1459)\n  - [x] `Value.geo_point_value` added (#1464)\n  - [x] `Value.blob_key_value` removed\n  - [x] `CompositeFilter.filter  --\u003e  CompositeFilter.filters` (~~#1356~~, #1454)\n  - [x] `CompositeFilter.operation  --\u003e  CompositeFilter.op` (~~#1356~~, #1454)\n  - [x] `PropertyFilter.operation  --\u003e  PropertyFilter.op` (~~#1356~~, #1454)\n  - [x] `CompositeFilter.AND  --\u003e  CompositeFilter.OPERATOR_UNSPECIFIED` (as default value)  (~~#1356~~, #1454)\n  - [x] `ReadOptions.DEFAULT  --\u003e  ReadOptions.READ_CONSISTENCY_UNSPECIFIED` (as default value)  (~~#1356~~, #1454)\n- [x] Deal with the fact that `Entity.property` is not a `map` called `Entity.properties` (somewhat different semantics) (#1340)\n- [x] Tear out dataset (project) [prefix code](https://github.com/GoogleCloudPlatform/gcloud-python/blob/3f189536262a7f2046a66c5fb0cb9487ecb0269c/gcloud/datastore/helpers.py#L421) (#1466)\n- [x] Re-vamp environment variable usage (`dataset_id` no longer needed?) (#1465)\n- [x] Remove use of `isolation_level` in `Connection.begin_transaction` (#1343, #1407)\n\nNew Features:\n- [ ] Add `GqlQuery` support (#304)\n- [ ] `QueryResultBatch.skipped_cursor` added (only set when `skipped_results != 0`)\n- [ ] `EntityResult.cursor` added (set by the backend when the entity result is part of a `RunQueryResponse.batch.entity_results` response)\n- [x] Parse protobuf [errors](https://github.com/GoogleCloudPlatform/gcloud-python/pull/1466#issuecomment-183822889)\n","author":{"url":"https://github.com/dhermes","@type":"Person","name":"dhermes"},"datePublished":"2015-12-15T17:47:59.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":26},"url":"https://github.com/1288/google-cloud-python/issues/1288"}