René's URL Explorer Experiment

Title: fix: Use LONGBLOB for SQL registry proto columns on MySQL by nikolauspschuetz · Pull Request #6566 · feast-dev/feast · GitHub

Open Graph Title: fix: Use LONGBLOB for SQL registry proto columns on MySQL by nikolauspschuetz · Pull Request #6566 · feast-dev/feast

X Title: fix: Use LONGBLOB for SQL registry proto columns on MySQL by nikolauspschuetz · Pull Request #6566 · feast-dev/feast

Description: What The SQL registry stores each Feast object as a serialized protobuf in a binary column. On MySQL/MariaDB, SQLAlchemy's LargeBinary maps to BLOB, which caps at 64 KB. A single FeatureView proto routinely exceeds that, so MySQL silently truncates the write and the registry later fails to load with a protobuf DecodeError (e.g. feast serve failing to start). PostgreSQL and SQLite were never affected. Fixes #5800. How Introduce a dialect-aware ProtoBytes type that emits LONGBLOB on MySQL and MariaDB while keeping LargeBinary's default mapping on every other dialect (BLOB on SQLite, BYTEA on PostgreSQL), and apply it to all binary proto/metadata columns. MariaDB is registered separately because SQLAlchemy 2.x reports dialect.name == "mariadb". The variants are chained (not variadic) so the expression also works on SQLAlchemy 1.4.x, which Feast still supports (SQLAlchemy>=1.4.19). metadata.create_all only creates missing tables, so existing MySQL registries are not migrated automatically. Added a best-effort startup check (_warn_if_narrow_blob_columns) that, on MySQL/MariaDB, logs an error naming any columns still typed BLOB and points at the documented migration. It is logged at ERROR (so monitoring pipelines that filter below ERROR still catch it) but deliberately does not refuse to start — a registry whose protos all fit in 64 KB is unaffected and a routine upgrade should not break it. (Happy to switch to fail-fast-with-escape-hatch if maintainers prefer.) Documented the migration in the SQL registry reference, including the metadata-lock risk, a mandatory "stop writers first" checklist, and online-schema-change guidance (pt-online-schema-change / gh-ost). Note: BLOB→LONGBLOB is a column type change, which InnoDB performs with ALGORITHM=COPY (rebuild) — INPLACE is not supported. Tests Unit (tests/unit/infra/registry/test_sql_registry.py): assert the compiled DDL emits LONGBLOB on MySQL/MariaDB, BLOB on SQLite, BYTEA on PostgreSQL; cover the startup-check paths (errors on stale, silent when migrated, skips non-MySQL, never raises on query failure, runs on both read+write engines). Integration (tests/integration/registration/test_universal_registry.py): test_apply_feature_view_large_proto_roundtrip_mysql applies a FeatureView with a >64 KB proto on the live mysql_registry / mysql_registry_async fixtures and reads it back from the DB (allow_cache=False), asserting byte-identity — the regression guard for the truncation bug. Docs / agent guidance docs/reference/registries/sql.md — new "serialized-proto columns use LONGBLOB" section + migration runbook. Updated the in-repo agent rules/skills (feast-components rule pair, feast-architecture and feast-testing SKILLs) so the ProtoBytes convention and test patterns are discoverable.

Open Graph Description: What The SQL registry stores each Feast object as a serialized protobuf in a binary column. On MySQL/MariaDB, SQLAlchemy's LargeBinary maps to BLOB, which caps at 64 KB. A single FeatureView pr...

X Description: What The SQL registry stores each Feast object as a serialized protobuf in a binary column. On MySQL/MariaDB, SQLAlchemy's LargeBinary maps to BLOB, which caps at 64 KB. A single FeatureVie...

Opengraph URL: https://github.com/feast-dev/feast/pull/6566

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width