René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feast push (Redshift/DynamoDb) not work with PushMode.ONLINE_AND_OFFLINE when more than 500 columns","articleBody":"## Expected Behavior \r\nCurrently, we have a push source with Redshift Offline Store and DynamoDb  Online Store. \r\nWe built our view with more than 500 columns. Around 750 columns. \r\n\r\nWe expected to ingest data in dynamo and redshift when we run \r\n`fs.push(\"push_source\", df, to=PushMode.ONLINE_AND_OFFLINE)`\r\n\r\n## Current Behavior\r\n\r\nPush command raise an issue like` [ERROR] ValueError: The input dataframe has columns ..`\r\nThis issue come from `get_table_column_names_and_types `method in `write_to_offline_store` method.\r\nIn the method, we check if `if set(input_columns) != set(source_columns)` and raise the below issue if there are diff. \r\n\r\nIn case with more than 500 columns we get a diff because source_columns come from `get_table_column_names_and_types` method result where the result is define by MaxResults parameters. \r\n\r\n## Steps to reproduce\r\n```\r\nentity= Entity(\r\n    name=\"entity\",\r\n    join_keys=[\"entity_id\"],\r\n    value_type=ValueType.INT64,\r\n)\r\n\r\npush_source = PushSource(\r\n    name=\"push_source\",\r\n    batch_source=RedshiftSource(\r\n        table=\"fs_push_view\",\r\n        timestamp_field=\"datecreation\",\r\n        created_timestamp_column=\"created_at\"),\r\n)\r\n\r\nbesoin_embedding_push_view = FeatureView(\r\n    name=\"push_view\",\r\n    entities=[entity],\r\n    schema=[Field(name=f\"field_{dim}\", dtype=types.Float64) for dim in range(768)],\r\n    source=push_source \r\n)\r\n\r\nfs.push(\"push_source\", df, to=PushMode.ONLINE_AND_OFFLINE)\r\n\r\n```\r\n\r\n### Specifications\r\n\r\n- Version: 0.25.0\r\n- Platform: AWS\r\n- Subsystem:\r\n\r\n## Possible Solution\r\n\r\nIn my mind, we have two solutions:\r\n\r\n- Set higher MaxResults in describe_table method\r\n- Use NextToken to iterate through results\r\n\r\n","author":{"url":"https://github.com/beubeu13220","@type":"Person","name":"beubeu13220"},"datePublished":"2022-10-10T14:06:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/3282/feast/issues/3282"}