René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Incorrect Forex Price Spikes","articleBody":"Forex Websocket prices (\"CAS.*\") and Agg Data have occasional price spikes.  These price spikes don't align with TradingView chart or Forex.com. \r\n\r\nI found the reason why:\r\nPolygon.io is using exchange bid price instead of trade price to populate websocket and agg data.\r\n\r\n\r\nHere I am printing prices of CADJPY. The price 112.735 is an outlier that never happened. That price came from a stink bid on the exchange.\r\n\r\nPrint outs of Aggs and Quotes for the surrounding timestamps:\r\n\r\n```\r\nagg Agg(open=113.7, high=113.7, low=113.7, close=113.7, volume=1, vwap=113.7, timestamp=1715288494000, transactions=1, otc=None) dt 8999 ms\r\nagg Agg(open=112.735, high=112.735, low=112.735, close=112.735, volume=1, vwap=112.735, timestamp=1715288495000, transactions=1, otc=None) dt 7999 ms\r\nagg Agg(open=112.735, high=112.735, low=112.735, close=112.735, volume=1, vwap=112.735, timestamp=1715288497000, transactions=1, otc=None) dt 5999 ms\r\nagg Agg(open=113.53, high=113.53, low=113.53, close=113.53, volume=1, vwap=113.53, timestamp=1715288500000, transactions=1, otc=None) dt 2999 ms\r\nagg Agg(open=113.682, high=113.682, low=113.682, close=113.682, volume=1, vwap=113.682, timestamp=1715288503000, transactions=1, otc=None) dt -1 ms\r\nagg Agg(open=112.735, high=112.735, low=112.735, close=112.735, volume=1, vwap=112.735, timestamp=1715288504000, transactions=1, otc=None) dt -1001 ms\r\nagg Agg(open=113.62, high=113.62, low=113.62, close=113.62, volume=1, vwap=113.62, timestamp=1715288505000, transactions=1, otc=None) dt -2001 ms\r\nagg Agg(open=113.7, high=113.7, low=113.7, close=113.7, volume=1, vwap=113.7, timestamp=1715288509000, transactions=1, otc=None) dt -6001 ms\r\nagg Agg(open=113.63, high=113.63, low=113.63, close=113.63, volume=1, vwap=113.63, timestamp=1715288510000, transactions=1, otc=None) dt -7001 ms\r\n\r\n```\r\n\r\n\r\n```\r\nquote Quote(ask_exchange=48, ask_price=113.66, ask_size=None, bid_exchange=48, bid_price=113.62, bid_size=None, conditions=None, indicators=None, participant_timestamp=1715288505000000000, sequence_number=None, sip_timestamp=None, tape=None, trf_timestamp=None)\r\nquote Quote(ask_exchange=48, ask_price=114.358, ask_size=None, bid_exchange=48, bid_price=112.735, bid_size=None, conditions=None, indicators=None, participant_timestamp=1715288504000000000, sequence_number=None, sip_timestamp=None, tape=None, trf_timestamp=None)\r\nquote Quote(ask_exchange=48, ask_price=113.805, ask_size=None, bid_exchange=48, bid_price=113.682, bid_size=None, conditions=None, indicators=None, participant_timestamp=1715288503000000000, sequence_number=None, sip_timestamp=None, tape=None, trf_timestamp=None)\r\nquote Quote(ask_exchange=48, ask_price=113.57, ask_size=None, bid_exchange=48, bid_price=113.53, bid_size=None, conditions=None, indicators=None, participant_timestamp=1715288500000000000, sequence_number=None, sip_timestamp=None, tape=None, trf_timestamp=None)\r\n\r\n\r\n```\r\n\r\n\r\nAttaching images of these crazy spikes when monitoring websocket data:\r\n\r\n\r\n![image](https://github.com/polygon-io/client-python/assets/161871533/1f0dd310-fcdf-406d-8ffd-1ebcad985eb6)\r\n\r\n![image](https://github.com/polygon-io/client-python/assets/161871533/fb4b2476-2778-4c40-94fd-94d0d90bff0c)\r\n\r\n\r\n![image](https://github.com/polygon-io/client-python/assets/161871533/770488f4-6e8d-4172-aa29-0b0533bb8411)\r\n\r\nPossible solutions:\r\n1. Get actual trade data from data providers and use this to populate websocket and agg data instead of faking it.\r\n\r\n2. If trade data is not available, i recommend averaging the bid and ask price instead of just using the bid price. If the delta between bid and ask is too high, discard the price point. This seems to produce reasonable results. \r\n\r\n\r\nWithout a solution, there seems to be no point to having the \"\"CAS.*\" and \"Agg\" data when it merely uses the bid price in the existing Quote data. I implemented solution #2 already in my branch if you want to reference it, Solution #1 is the proper and preferred fix tho","author":{"url":"https://github.com/jbonilla-tao","@type":"Person","name":"jbonilla-tao"},"datePublished":"2024-05-10T23:12:47.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/668/client-python/issues/668"}