René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add `date_partition_column` to `SparkSource`","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nThe current spark implementation scans over all parquet files. This process can be made faster and more efficient by specifying a `date_partition_column`. During execution, this column would be used to filter the data at a file level. Only files who's date is within the range would be scanned.\r\n\r\n**Describe the solution you'd like**\r\nAdd `date_partition_column` to `SparkSource`. A similar implementation exists for the `AthenaSource`\r\n\r\n**Describe alternatives you've considered**\r\nNone\r\n\r\nI have implemented this locally and it works. I'm happy to open a PR\r\n","author":{"url":"https://github.com/niklasvm","@type":"Person","name":"niklasvm"},"datePublished":"2024-12-11T09:53:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/4835/feast/issues/4835"}