René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"No support for null UnixTimestamp","articleBody":"## Expected Behavior \r\n\r\nThis script should return online features, with one null in `last_purchased_date`\r\n```python\r\nfrom feast import Entity, FeatureView\r\nfrom feast.infra.offline_stores.file_source import FileSource\r\nfrom feast.repo_config import RegistryConfig, RepoConfig\r\nfrom feast import FeatureStore\r\nfrom datetime import datetime\r\nfrom feast.types import Int32, UnixTimestamp\r\nfrom feast import Field\r\n\r\nimport pandas as pd\r\n\r\n\r\n# create dataset\r\npd.DataFrame([\r\n    {\"user_id\": 1, \"event_timestamp\": datetime(2022, 5, 1), \"created\": datetime(2022, 5, 1), \"purchases\": 3, \"last_purchase_date\": datetime(2022, 4, 23, 13, 4, 1)},\r\n    {\"user_id\": 2, \"event_timestamp\": datetime(2022, 5, 2), \"created\": datetime(2022, 5, 2), \"purchases\": 1, \"last_purchase_date\": datetime(2022, 2, 1, 11, 4, 1)},\r\n    {\"user_id\": 3, \"event_timestamp\": datetime(2022, 5, 2), \"created\": datetime(2022, 5, 2), \"purchases\": 0, \"last_purchase_date\": None},    \r\n]).to_parquet('user_stats.parquet')\r\n\r\n\r\nuser = Entity(name=\"user_id\", description=\"user id\")\r\n\r\nuser_stats_view = FeatureView(\r\n    name=\"user_stats\",\r\n    entities=[user],\r\n    source=FileSource(\r\n        path=\"user_stats.parquet\",\r\n        timestamp_field=\"event_timestamp\",\r\n        created_timestamp_column=\"created\",\r\n    ),\r\n    schema=[\r\n        Field(name=\"purchases\", dtype=Int32),\r\n        Field(name=\"last_purchase_date\", dtype=UnixTimestamp),\r\n    ]\r\n)\r\n\r\nonline_store_path = 'online_store.db'\r\nregistry_path = 'registry.db'\r\n\r\nrepo = RepoConfig(\r\n    registry=\"registry.db\",\r\n    project='feature_store',\r\n    provider=\"local\",\r\n    offline_store=\"file\",\r\n    use_ssl=True, \r\n    is_secure=True,\r\n    validate=True,\r\n)\r\n\r\nfs = FeatureStore(config=repo)\r\n\r\nfs.apply([user, user_stats_view])\r\n\r\nfs.materialize_incremental(end_date=datetime.utcnow())\r\n\r\n\r\nentity_rows = [{\"user_id\": i} for i in range(1, 4)]\r\n\r\n\r\nfeature_df = fs.get_online_features(\r\n    features=[\r\n        \"user_stats:purchases\",\r\n        \"user_stats:last_purchase_date\",\r\n    ],\r\n    entity_rows=entity_rows\r\n).to_df()\r\nprint(feature_df)\r\n```\r\n\r\n## Current Behavior\r\n\r\n```\r\nMaterializing 1 feature views to 2022-06-16 20:30:41-06:00 into the sqlite online store.\r\n\r\nSince the ttl is 0 for feature view user_stats, the start date will be set to 1 year before the current time.\r\nuser_stats from 2021-06-17 20:30:41-06:00 to 2022-06-16 20:30:41-06:00:\r\n100%|████████████████████████████████████████████████████████████████| 3/3 [00:00\u003c00:00, 420.36it/s]\r\nTraceback (most recent call last):\r\n  File \"null_timestamp_example.py\", line 59, in \u003cmodule\u003e\r\n    feature_df = fs.get_online_features(\r\n  File \"/Users/apope/feast/sdk/python/feast/online_response.py\", line 79, in to_df\r\n    return pd.DataFrame(self.to_dict(include_event_timestamps))\r\n  File \"/Users/apope/feast/sdk/python/feast/online_response.py\", line 59, in to_dict\r\n    response[feature_ref] = [\r\n  File \"/Users/apope/feast/sdk/python/feast/online_response.py\", line 60, in \u003clistcomp\u003e\r\n    feast_value_type_to_python_type(v) for v in feature_vector.values\r\n  File \"/Users/apope/feast/sdk/python/feast/type_map.py\", line 74, in feast_value_type_to_python_type\r\n    val = datetime.fromtimestamp(val, tz=timezone.utc)\r\nOSError: [Errno 84] Value too large to be stored in data type\r\n```\r\n\r\n## Steps to reproduce\r\n\r\n### Specifications\r\n\r\n- Version: 0.21.2\r\n- Platform: Mac\r\n- Subsystem: \r\n\r\n## Possible Solution\r\n\r\nThis happens because, while materializing, in `_python_datetime_to_int_timestamp()` the `NaT` value gets converted to `-9223372036854775808`\r\n\r\n```python\r\nIn [108]:\r\nfrom typing import cast, Sequence\r\nimport numpy as np\r\n\r\ncast(Sequence[np.int_], np.array(['nat'], dtype='datetime64[ns]').astype('datetime64[s]').astype(np.int_))\r\nOut [108]:\r\narray([-9223372036854775808])\r\n```\r\n\r\nWhich is out of range for `datetime.fromtimestamp()`:\r\n```python\r\nIn [109]:\r\nfrom datetime import datetime, timezone\r\n\r\ndatetime.fromtimestamp(-9223372036854775808, tz=timezone.utc)\r\n\r\nTruncated Traceback (Use C-c C-$ to view full TB):\r\n/tmp/ipykernel_58/1143168205.py in \u003cmodule\u003e\r\n      1 from datetime import datetime, timezone\r\n      2 \r\n----\u003e 3 val = datetime.fromtimestamp(-9223372036854775808, tz=timezone.utc)\r\n\r\nOSError: [Errno 75] Value too large for defined data type\r\n```\r\n\r\nA simple fix would be to leave the materialization logic as-is and, when deserializing in `feast_value_type_to_python_type()`, just catch this one value and return a null instead.","author":{"url":"https://github.com/aapope","@type":"Person","name":"aapope"},"datePublished":"2022-06-16T21:14:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/2803/feast/issues/2803"}