René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add client-side batching for CreateMultiple/UpdateMultiple/UpsertMultiple","articleBody":"## Problem\n\n`CreateMultiple`, `UpdateMultiple`, and `UpsertMultiple` each send **all records in a single POST** regardless of count. There is no client-side chunking.\n\n- `_create_multiple` (data/_odata.py:316-376) — builds one `{\"Targets\": [...]}` payload with every record and POSTs it.\n- `_update_multiple` (data/_odata.py:656-697) — same pattern.\n- `_upsert_multiple` (data/_odata.py:440-493) — same pattern.\n\nDataverse has a server-side limit (typically 1,000 records per `*Multiple` call). Sending more can result in 400/413 errors or timeouts. Today, callers must chunk manually in their scripts. The SDK should handle this internally.\n\n## Proposed changes\n\n### 1. Client-side batching (correctness fix)\n\nSplit large record lists into 1,000-record chunks and send each as a separate POST. This is the minimum viable fix.\n\n```python\n# Pseudocode for _create_multiple with batching\nBATCH_SIZE = 1000\n\ndef _create_multiple(self, entity_set, table_schema_name, records):\n    all_ids = []\n    for i in range(0, len(records), BATCH_SIZE):\n        chunk = records[i:i + BATCH_SIZE]\n        ids = self._create_multiple_batch(entity_set, table_schema_name, chunk)\n        all_ids.extend(ids)\n    return all_ids\n```\n\n**Atomicity trade-off:** Today a single POST is atomic (all-or-nothing). Splitting into batches means partial success is possible — batch 1 succeeds, batch 2 fails, leaving the caller with a partial import. This should be clearly documented. Callers who need atomicity should limit their input to \u003c=1000 records.\n\n### 2. Optional concurrent batch dispatch (performance, follow-on)\n\nAfter batching exists, add an opt-in `max_workers` parameter to dispatch batches concurrently via `concurrent.futures.ThreadPoolExecutor` (stdlib, no new dependency).\n\n```python\ndef create(self, table, data, *, max_workers=1):\n    # max_workers=1 (default) = sequential, identical to today\n    # max_workers=4 = 4 concurrent batch POSTs\n```\n\n**Default must be 1** (sequential) to avoid any regression:\n- No extra threads on slow machines\n- No extra memory overhead\n- No concurrent request spike hitting Dataverse rate limits\n- Identical behavior to today unless user explicitly opts in\n\nWhen `max_workers \u003e 1`:\n- Uses ThreadPoolExecutor (~8MB stack per thread, bounded by `max_workers`)\n- Respects 429 (rate limit) responses — backs off all workers\n- Connection pooling via existing `_HttpClient` session support\n\n### 3. Page pre-fetching in `_get_multiple` (separate enhancement)\n\n`_get_multiple` (data/_odata.py:821-826) fetches pages sequentially in a `while next_link` loop. Each page blocks until complete before the next is requested.\n\nPre-fetching 1 page ahead while the caller processes the current page would overlap I/O with processing:\n\n```python\ndef _get_multiple(self, ..., prefetch_pages=0):\n    # prefetch_pages=0 (default) = sequential, identical to today\n    # prefetch_pages=1 = fetch next page while caller processes current\n```\n\n**Default must be 0** to avoid buffering extra pages in memory. A single pre-fetched page for a 5,000-record default page size is ~5-20MB depending on column count — acceptable when opted in, but shouldn't be forced.\n\n### 4. Picklist cache warming (separate enhancement)\n\n`_optionset_map` (data/_odata.py:1219-1331) makes 2 HTTP calls per string field on cache miss. The cache works well for subsequent records, but the first record with N string fields triggers 2N sequential HTTP calls.\n\nA `warm_picklist_cache(table)` method that fetches all picklist metadata for a table in a single request would eliminate the cold-start penalty for bulk operations.\n\n## APIs NOT proposed for parallelism\n\n| API | Why not |\n|---|---|\n| Chunked file upload (`_upload.py:117-195`) | Protocol is sequential by design — uses session token with `Content-Range` headers, each chunk returns 206 before next can be sent |\n| Column creation (`_odata.py:1712-1762`) | Dataverse metadata locks on the same table can cause conflicts with concurrent POSTs |\n| Column deletion (`_odata.py:1764-1831`) | Same metadata lock concern |\n| Relationship creation (`_relationships.py`) | Same metadata lock concern |\n| BulkDelete (`_odata.py:548-618`) | Already async server-side; splitting into concurrent jobs adds complexity with minimal benefit |\n\n## Context\n\nIdentified during end-to-end validation of a 21-table dataset import. The agent-generated script had to implement its own chunking (`chunk_size=1000`) because the SDK doesn't handle it. Client-side batching should be an SDK responsibility, not something every caller reinvents.","author":{"url":"https://github.com/suyask-msft","@type":"Person","name":"suyask-msft"},"datePublished":"2026-04-05T21:18:09.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/156/PowerPlatform-DataverseClient-Python/issues/156"}