René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Parquet Schema Inference only supports File, not directory","articleBody":"When using a FileSource that is in Parquet format, if the source happens to be a directory of partitioned Parquet files, the following lines throw an error:\r\n\r\nhttps://github.com/feast-dev/feast/blob/01d3568168bb9febb9fbda4988283b3886c32a31/sdk/python/feast/infra/offline_stores/file_source.py#L182-L184\r\n\r\n`OSError: Expected file path, but /home/ubuntu/project/data/driver_stats_partitioned is a directory`\r\n\r\nHow to replicate:\r\n\r\n1. Start with a demo feast project (`feast init`)\r\n2. Create a partitioned Parquet Dataset.  Use the following to create a dataset with only a single timestamp for inference\r\n```\r\nimport pyarrow.parquet as pq\r\ndf = pq.read_table(\"./data/driver_stats.parquet\")\r\ndf = df.drop([\"created\"])\r\npq.write_to_dataset(df, \"./data/driver_stats_partitioned\")\r\n```\r\n3. Update the file source in `example.py` to look like this:\r\n```\r\ndriver_hourly_stats = FileSource(\r\n    path=\"/home/ubuntu/cado-feast/feature_store/exciting_sunbeam/data/driver_stats_partitioned2\",\r\n)\r\n```\r\n\r\n4. Run `feast apply`\r\nFor now, I've been able to fix by updating the above lines to:\r\n```\r\nschema = ParquetDataset(\r\n    path if filesystem is None else filesystem.open_input_file(path)\r\n).schema.to_arrow_schema()\r\n```","author":{"url":"https://github.com/dvanbrug","@type":"Person","name":"dvanbrug"},"datePublished":"2022-05-13T19:56:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/2685/feast/issues/2685"}