René's URL Explorer Experiment

Title: feat(aicore/fallback): add opt-in model fallback for Orchestration v2 by lenin-ribeiro · Pull Request #185 · SAP/cloud-sdk-python · GitHub

Open Graph Title: feat(aicore/fallback): add opt-in model fallback for Orchestration v2 by lenin-ribeiro · Pull Request #185 · SAP/cloud-sdk-python

X Title: feat(aicore/fallback): add opt-in model fallback for Orchestration v2 by lenin-ribeiro · Pull Request #185 · SAP/cloud-sdk-python

Description: Disclaimer: Do not include SAP-internal or customer-specific information in this PR (e.g. internal system URLs, customer names, tenant IDs, or confidential configurations). This is a public repository. Description Adds opt-in model fallback for SAP AI Core Orchestration v2 to the sap_cloud_sdk.aicore module. Orchestration v2 supports preference-ordered fallback module configurations: when the primary call fails (model unsupported in region, 429, 408, or any 5xx — and unsupported-model only for streaming), the server transparently retries with the next preference. The underlying litellm SAP provider already builds body["config"]["modules"] as a list when fallback_sap_modules is present in optional_params; what was missing was the SDK-side ergonomic surface and the response-side visibility into which preferences were skipped. This PR introduces: FallbackModel, FallbackConfig — typed dataclasses for declaring per-preference model + params + version. set_fallbacks(config) — single entry point mirroring set_filtering(). Fallback is opt-in: set_aicore_config() does NOT activate it. Developers either call set_fallbacks(...) programmatically or set AICORE_FALLBACK_ENABLED=true (with AICORE_FALLBACK_MODELS or AICORE_FALLBACK_CONFIG) and call set_fallbacks() with no args. response.intermediate_failures — when the fallback path fires, the per-preference failure list from the orchestration response is surfaced as an attribute on the returned ModelResponse. None when the primary succeeded, useful as a quick check. OrchestrationPatchConfig — a subclass of FilteringOrchestrationConfig that adds the fallback-side transform_request / transform_response hooks. Filtering stays entirely in the filtering/ package; the fallback subclass reuses filtering behaviour through inheritance. Filtering broadcast across fallbacks — when both filtering and fallback are active, the filtering config is applied to every module entry on the wire (previously modules[0] only). Consistent SDK-side default; if a fallback should run unfiltered, the developer can call disable_filtering() before the call. Related Issue N/A — additive feature, no issue tracked Type of Change Bug fix (non-breaking change that fixes an issue) New feature (non-breaking change that adds functionality) Breaking change (fix or feature that would cause existing functionality to change) Documentation update How to Test Unit tests (no live credentials required) uv run python -m pytest tests/aicore -v Expect 153 passed, 8 skipped (the 8 skips are integration scenarios waiting for live env vars). Integration tests (requires AI Core access) Copy .env_integration_tests.example to .env_integration_tests and fill in the AI Core creds. Set the new fallback secrets: AICORE_FALLBACK_TEST_PRIMARY_MODEL=sap/this-model-does-not-exist AICORE_FALLBACK_TEST_FALLBACK_MODEL=sap/mistralai--mistral-small-instruct The primary should be a model name the orchestration server reports as unsupported in your region — a genuinely nonexistent name works and avoids depending on transient 5xx errors. Run: uv run python -m pytest tests/aicore/integration/test_fallback_bdd.py -v Expected: 4 scenarios pass (primary succeeds, primary unsupported → fallback used, filtering + fallback composition, streaming + fallback). Manual smoke from sap_cloud_sdk.aicore import ( FallbackConfig, FallbackModel, completion, set_aicore_config, set_fallbacks, ) set_aicore_config() set_fallbacks(FallbackConfig([ FallbackModel(model="sap/mistralai--mistral-small-instruct"), ])) response = completion( model="sap/gpt-4o", # or any potentially-unsupported model messages=[{"role": "user", "content": "Translate 'hello' to German."}], ) print(response.choices[0].message.content) print(getattr(response, "intermediate_failures", None)) # None if primary worked; list of failure dicts if fallback fired. Checklist I have read the Contributing Guidelines I have verified that my changes solve the issue I have added/updated automated tests to cover my changes All tests pass locally (pytest tests/aicore, ruff check, ruff format --check, ty check) I have verified that my code follows the Code Guidelines I have updated documentation (if applicable) — aicore/user-guide.md gains a "Model Fallback (opt-in)" section I have added type hints for all public APIs My code does not contain sensitive information (credentials, tokens, etc.) I have followed Conventional Commits for commit messages Breaking Changes None for users of sap_cloud_sdk.aicore public APIs. There is one user-visible behavioural change that only affects users who use filtering AND fallback together (an impossible combination on main today since fallback didn't exist): when both are active, the filtering configuration now applies to every module entry (primary + every fallback), not just modules[0]. This is the safe-by-default semantic — to run a fallback unfiltered, explicitly disable_filtering() before that call. Documented in the user guide. Additional Notes Design choices Fallback lives in its own package (src/sap_cloud_sdk/aicore/fallback/), mirroring how filtering lives in filtering/. The public API — FallbackModel, FallbackConfig, set_fallbacks — is flat-re-exported from sap_cloud_sdk.aicore so callers don't touch the subpackage directly. OrchestrationPatchConfig inherits from FilteringOrchestrationConfig. One class handles both concerns at request time; filtering behaviour is reused, not reimplemented. When only filtering is active, FilteringOrchestrationConfig is the installed class exactly as on main; when fallback is installed, our subclass takes the slot. Filtering stays as it merged in #174. The only cross-module coupling is a small hook inside filtering._patch._install that defers to fallback when its config is active — so set_filtering / disable_filtering toggles while fallback is on update the filtering state without clobbering OrchestrationPatchConfig. Lazy import avoids a circular dependency. Env vars opt-in. AICORE_FALLBACK_ENABLED defaults to false (unlike filtering, which is on by default after set_aicore_config()). Two-tier schema: AICORE_FALLBACK_MODELS (comma list, simple case) + AICORE_FALLBACK_CONFIG (JSON, full per-model config; takes precedence). intermediate_failures on the response object. Pydantic ModelResponse uses extra="allow", so we can attach the field directly. Accessed via getattr(response, "intermediate_failures", None). v1 limitations (documented) intermediate_failures is surfaced for non-streaming responses only. Capturing the field from SAPStreamIterator chunks requires deeper changes to litellm internals and is deferred to a future iteration. The streaming integration test asserts that fallback still fires correctly server-side; it doesn't assert intermediate_failures. Telemetry Added Operation.AICORE_SET_FALLBACKS = "set_fallbacks". The set_fallbacks entry point is decorated with @record_metrics(Module.AICORE, Operation.AICORE_SET_FALLBACKS) per docs/GUIDELINES.md. Files added / changed NEW: src/sap_cloud_sdk/aicore/fallback/__init__.py src/sap_cloud_sdk/aicore/fallback/_patch.py src/sap_cloud_sdk/aicore/fallback/fallback.py src/sap_cloud_sdk/aicore/fallback/py.typed tests/aicore/fallback/__init__.py tests/aicore/fallback/unit/__init__.py tests/aicore/fallback/unit/test_fallback_config.py tests/aicore/fallback/unit/test_patch.py tests/aicore/fallback/unit/test_set_fallbacks.py tests/aicore/integration/fallback.feature tests/aicore/integration/test_fallback_bdd.py MODIFIED: src/sap_cloud_sdk/aicore/__init__.py # export FallbackModel / FallbackConfig / set_fallbacks src/sap_cloud_sdk/aicore/filtering/_patch.py # small hook: defer to fallback in _install when fallback is active src/sap_cloud_sdk/aicore/user-guide.md # new "Model Fallback (opt-in)" section src/sap_cloud_sdk/core/telemetry/operation.py # AICORE_SET_FALLBACKS tests/aicore/integration/conftest.py # fallback fixtures + clean-skip behaviour tests/core/unit/telemetry/test_operation.py # expected enum count 151 → 152 .env_integration_tests.example # AICORE_FALLBACK_TEST_PRIMARY_MODEL / _FALLBACK_MODEL

Open Graph Description: Disclaimer: Do not include SAP-internal or customer-specific information in this PR (e.g. internal system URLs, customer names, tenant IDs, or confidential configurations). This is a public reposit...

X Description: Disclaimer: Do not include SAP-internal or customer-specific information in this PR (e.g. internal system URLs, customer names, tenant IDs, or confidential configurations). This is a public reposit...

Opengraph URL: https://github.com/SAP/cloud-sdk-python/pull/185

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width