René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"First-class ingestion lineage, separate from the FK graph","articleBody":"## Summary\n\nRoadmap capability from the strict-provenance adoption analysis ([#1474 discussion](https://github.com/datajoint/datajoint-python/pull/1474#issuecomment-4849784492), @ttngu207): a **first-class record of where data came from, kept separate from the FK graph**.\n\n## Why\n\nDataJoint has one graph, used for two different things:\n- *what is derived from what* — durable, invariant, belongs in the FK graph;\n- *how a row arrived* — transient, operational (a file today, an API next quarter, a sync after that).\n\nForcing the second into the FK graph (the \"landing-table\" pattern: entities downstream of a `ParsedFile` ancestor) marries the slow-moving domain model to fast-moving infrastructure — every ingestion change becomes a schema migration, and historical rows permanently carry whichever ingestion ancestor existed at creation time. That is why the strict-provenance write boundary has no clean answer for the common **fat ingestion `make()`** (one file parsed into many top-level entity tables): every compliant restructuring corrupts the domain model or abandons DataJoint orchestration.\n\n## Shape\n\n- **Append-only ingestion-lineage records**, per row or per insert batch: `(entity key, source descriptor, timestamp, actor)` — multiple sources over an entity's lifetime, zero schema changes.\n- Queryable alongside (not through) the FK graph; surfaced in row-lineage views.\n- Candidate capture points: the gated insert path (already intercepts every write under strict mode), `staged_insert1`, and populate job metadata.\n\n## Related use cases to fold in\n\n- Spyglass **Exports** (discussion #1232, @CBroz1): monitor `fetch`/`restrict`/`join` calls, log accessed leaves, drive an upward trace for `mysqldump`/DANDI publication — the same \"record what was touched, separate from the FK graph\" mechanism from the read side.\n- Platform row-lineage / compute-provenance work (branch identity ↔ per-row code-version tagging) — ingestion lineage is the missing third leg beside code-version and FK derivation.\n","author":{"url":"https://github.com/dimitri-yatsenko","@type":"Person","name":"dimitri-yatsenko"},"datePublished":"2026-07-04T20:40:39.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/1483/datajoint-python/issues/1483"}