{"id":278,"date":"2026-10-01T15:35:02","date_gmt":"2026-10-01T07:35:02","guid":{"rendered":"https:\/\/blog.infoway.io\/en\/?p=278"},"modified":"2026-10-01T15:35:05","modified_gmt":"2026-10-01T07:35:05","slug":"forex-api-in-production-rest-for-backtesting-websocket-for-live-data","status":"publish","type":"post","link":"https:\/\/blog.infoway.io\/en\/forex-api-in-production-rest-for-backtesting-websocket-for-live-data\/","title":{"rendered":"Forex API in Production: REST for Backtesting, WebSocket for Live Data"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Most forex data problems in production don&#8217;t come from bad data. They come from using the right data through the wrong interface: a backtest that polls a live endpoint, a dashboard that refreshes by hammering REST every second, or a live strategy that silently trades on a feed that dropped for four minutes without anyone noticing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The rule of thumb is short: <strong>use a REST API for backtesting and historical work, and a WebSocket API for real-time forex streaming.<\/strong> The useful part is everything around that rule: how to page through three years of minute bars without hitting rate limits, why your backtest needs spread data that historical candles don&#8217;t contain, and how to repair the hole a dropped connection leaves in your live data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This guide walks through that architecture end to end, with Python examples against the <a href=\"https:\/\/infoway.io\/en\/forex-api\" target=\"_blank\" rel=\"noopener\">Infoway Forex API<\/a>. Each pattern carries over to other providers too.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What you&#8217;ll build:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A rate-limit-aware historical backfill that pages backwards through 1-minute candles<\/li>\n\n\n\n<li>Validation checks that understand the forex weekly session (so weekends aren&#8217;t flagged as data loss)<\/li>\n\n\n\n<li>A spread model for backtests, built from live bid\/ask data<\/li>\n\n\n\n<li>A resilient WebSocket consumer with heartbeats, backoff, and automatic gap repair over REST<\/li>\n\n\n\n<li>One canonical data model shared by the research and live paths<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Why the Workload Decides the Interface<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The two interfaces solve different problems:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th><\/th><th>Historical \/ research work<\/th><th>Live \/ production work<\/th><\/tr><\/thead><tbody><tr><td>Shape of the request<\/td><td>Bounded: one pair, one time range, one result set<\/td><td>Unbounded: updates keep arriving until you disconnect<\/td><\/tr><tr><td>What matters most<\/td><td>Repeatability, completeness, auditability<\/td><td>Freshness, low latency, staying connected<\/td><\/tr><tr><td>Failure mode<\/td><td>A request fails; retry it<\/td><td>The connection drops; data during the outage is gone<\/td><\/tr><tr><td>Right interface<\/td><td><strong>REST<\/strong><\/td><td><strong>WebSocket<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A REST request is a question with a definite answer: &#8220;give me 500 one-minute EURUSD bars ending at this timestamp.&#8221; You can rerun it tomorrow, cache it, checkpoint it, and log exactly what you asked for. That&#8217;s what research needs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A <a href=\"https:\/\/docs.infoway.io\/en-docs\/websocket\/subscription-address\" target=\"_blank\" rel=\"noopener\">WebSocket<\/a> connection is a subscription. The server pushes each update as it happens, without the latency and quota cost of polling. That&#8217;s what dashboards, alerts, signal engines, and risk monitors need.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Production systems need both, plus a defined way for the two to cover for each other.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One more distinction to settle up front: <strong>central-bank reference rates are not market data.<\/strong> The ECB&#8217;s daily euro reference rates are fine for accounting, invoicing, and month-end valuation, but they&#8217;re one fixing per day with no bid, no ask, and no intraday movement. Don&#8217;t backtest a trading strategy on them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Part 1: REST for Backtesting<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Know what the historical endpoint actually returns<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Before writing any backfill code, pin down three things about your provider&#8217;s historical data, because each one changes your design:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Granularity.<\/strong> Is the history in ticks or in candles? Infoway&#8217;s REST history is candle-based: 1-minute up to yearly bars through <code>POST \/common\/v2\/batch_kline<\/code> (the <code>common<\/code> path covers forex, metals, energy, and futures). Minute-level history goes back about three years; daily and higher intervals have no lookback limit.<\/li>\n\n\n\n<li><strong>Paging model.<\/strong> Infoway returns up to <strong>500 bars per request for a single symbol<\/strong>, ending at the <code>timestamp<\/code> you pass (in seconds). To go further back, you move that timestamp backwards. Multi-symbol requests return only the latest 2 bars per symbol, so do historical backfills <strong>one pair at a time<\/strong>.<\/li>\n\n\n\n<li><strong>Ordering and types.<\/strong> Bars come back <strong>newest first<\/strong>, and prices are <strong>strings<\/strong>. Both will bite you if you assume otherwise.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Budget the backfill before you run it<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Forex trades roughly 24 hours a day, 5 days a week, so one pair produces about 7,200 one-minute bars per week. Three years of 1-minute EURUSD is around <strong>1.1 million bars, or about 2,250 requests<\/strong> at 500 bars each.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Plan<\/th><th>REST rate limit<\/th><th>Time to backfill 3 years of 1-min data, one pair<\/th><\/tr><\/thead><tbody><tr><td>Free<\/td><td>1 request\/sec<\/td><td>~38 minutes<\/td><\/tr><tr><td>Basic<\/td><td>2 requests\/sec<\/td><td>~19 minutes<\/td><\/tr><tr><td>Advanced<\/td><td>10 requests\/sec<\/td><td>~4 minutes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A 10-pair research universe on the free plan is a six-hour job, which is fine as an overnight run but not something to restart from zero every time a request fails. That&#8217;s why the backfill below is <strong>checkpointed and idempotent<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">A backfill that pages backwards safely<\/h3>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>import json\nimport os\nimport time\nfrom decimal import Decimal\nfrom pathlib import Path\n\nimport requests\n\nAPI_KEY = os.environ&#91;\"INFOWAY_API_KEY\"&#93;\nBASE_URL = \"https:\/\/data.infoway.io\"\nHEADERS = {\"apiKey\": API_KEY, \"Content-Type\": \"application\/json\"}\n\nREQUEST_INTERVAL = 1.0          # seconds between calls; match your plan's limit\nRAW_DIR = Path(\"raw\/klines\")    # keep untouched responses for audit\/replay\n\n\ndef fetch_klines(pair: str, kline_type: int, num: int, end_ts: int | None = None) -> list&#91;dict&#93;:\n    \"\"\"One REST call: up to `num` bars for one pair, ending at end_ts (seconds).\"\"\"\n    body = {\"klineType\": kline_type, \"klineNum\": num, \"codes\": pair}\n    if end_ts is not None:\n        body&#91;\"timestamp\"&#93; = end_ts\n\n    for attempt in range(5):\n        resp = requests.post(f\"{BASE_URL}\/common\/v2\/batch_kline\",\n                             json=body, headers=HEADERS, timeout=15)\n        payload = resp.json() if resp.content else {}\n        if resp.status_code == 429 or payload.get(\"ret\") == 429:\n            time.sleep(2 ** attempt)          # back off and retry\n            continue\n        resp.raise_for_status()\n        if payload.get(\"ret\") != 200:\n            raise RuntimeError(f\"{pair}: ret={payload.get('ret')} msg={payload.get('msg')} \"\n                               f\"traceId={payload.get('traceId')}\")\n\n        # Persist the raw response before touching it\n        RAW_DIR.mkdir(parents=True, exist_ok=True)\n        (RAW_DIR \/ f\"{pair}_{kline_type}_{end_ts}.json\").write_text(json.dumps(payload))\n\n        data = payload.get(\"data\") or []\n        return data&#91;0&#93;&#91;\"respList\"&#93; if data else []\n    raise RuntimeError(f\"{pair}: still rate limited after retries\")\n\n\ndef backfill(pair: str, start_ts: int, end_ts: int, kline_type: int = 1) -> list&#91;dict&#93;:\n    \"\"\"Walk backwards from end_ts to start_ts, 500 bars at a time.\"\"\"\n    bars: dict&#91;int, dict&#93; = {}                  # keyed by bar time -> dedupes overlaps\n    cursor = end_ts\n\n    while cursor > start_ts:\n        batch = fetch_klines(pair, kline_type, 500, cursor)\n        time.sleep(REQUEST_INTERVAL)\n        if not batch:\n            break\n\n        for b in batch:\n            t = int(b&#91;\"t\"&#93;)\n            if start_ts &lt;= t &lt;= end_ts:\n                bars&#91;t&#93; = {\n                    \"pair\": pair,\n                    \"interval\": \"1m\",\n                    \"ts\": t,\n                    \"open\": Decimal(b&#91;\"o\"&#93;),\n                    \"high\": Decimal(b&#91;\"h\"&#93;),\n                    \"low\": Decimal(b&#91;\"l\"&#93;),\n                    \"close\": Decimal(b&#91;\"c\"&#93;),\n                    \"source\": \"rest\",\n                }\n\n        oldest = min(int(b&#91;\"t\"&#93;) for b in batch)\n        if oldest >= cursor:                    # no progress: stop instead of looping forever\n            break\n        cursor = oldest - 1\n        # save_checkpoint(pair, cursor)  -> resume from here after a crash\n\n    return [bars&#91;t&#93; for t in sorted(bars)]      # API returns newest-first; store oldest-first<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">json<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">os<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">time<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">from<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">decimal<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">from<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pathlib<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Path<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">requests<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">API_KEY<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">os.environ&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">INFOWAY_API_KEY<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">BASE_URL<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">https:\/\/data.infoway.io<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">HEADERS<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">apiKey<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">API_KEY,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Content-Type<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">application\/json<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">}<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">REQUEST_INTERVAL<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1.0<\/span><span style=\"color: #D8DEE9FF\">          <\/span><span style=\"color: #616E88\"># seconds between calls; match your plan&#39;s limit<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">RAW_DIR<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Path<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;raw\/klines&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># keep untouched responses for audit\/replay<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fetch_klines<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">kline_type:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">num:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">end_ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">|<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">None<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">list&#91;dict&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">&quot;&quot;&quot;One REST call: up to <\/span><span style=\"color: #ECEFF4\">`<\/span><span style=\"color: #88C0D0\">num<\/span><span style=\"color: #ECEFF4\">`<\/span><span style=\"color: #88C0D0\"> bars for one pair, ending at end_ts (seconds).&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">body<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">klineType<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">kline_type,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">klineNum<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">num,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">codes<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">end_ts<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">is<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">not<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">body&#91;<\/span><span style=\"color: #88C0D0\">&quot;timestamp&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">end_ts<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">attempt<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> range<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">5<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">resp<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">requests.post<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">f<\/span><span style=\"color: #88C0D0\">&quot;{BASE_URL}\/common\/v2\/batch_kline&quot;<\/span><span style=\"color: #88C0D0\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                             <\/span><span style=\"color: #D8DEE9\">json<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #A3BE8C\">body,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">headers<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #A3BE8C\">HEADERS,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">timeout<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">15<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">payload<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">resp.json<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">resp.content<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">else<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">resp.status_code<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">429<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">or<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">payload.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;ret&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">429<\/span><span style=\"color: #A3BE8C\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            time.sleep<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">2<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">**<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">attempt<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">          <\/span><span style=\"color: #616E88\"># back off and retry<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">continue<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">resp.raise_for_status<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        if payload.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;ret&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">!<\/span><span style=\"color: #D8DEE9FF\">= <\/span><span style=\"color: #B48EAD\">200<\/span><span style=\"color: #A3BE8C\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">raise<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">RuntimeError<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">f<\/span><span style=\"color: #88C0D0\">&quot;{pair}: ret={payload.get(&#39;ret&#39;)} msg={payload.get(&#39;msg&#39;)} &quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                               <\/span><span style=\"color: #88C0D0\">f<\/span><span style=\"color: #88C0D0\">&quot;traceId={payload.get(&#39;traceId&#39;)}&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># Persist the raw response before touching it<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">RAW_DIR.mkdir(parents<\/span><span style=\"color: #D8DEE9FF\">=True, <\/span><span style=\"color: #A3BE8C\">exist_ok=True<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">RAW_DIR<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">\/<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">f<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">{pair}_{kline_type}_{end_ts}.json<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #88C0D0\">.write_text(json.dumps(payload<\/span><span style=\"color: #D8DEE9FF\">))<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">data<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">payload.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;data&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">or<\/span><span style=\"color: #D8DEE9FF\"> []<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data&#91;<\/span><span style=\"color: #B48EAD\">0<\/span><span style=\"color: #A3BE8C\">&#93;&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">respList<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">else<\/span><span style=\"color: #D8DEE9FF\"> []<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">raise<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">RuntimeError<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">f<\/span><span style=\"color: #88C0D0\">&quot;{pair}: still rate limited after retries&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">backfill<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">start_ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">end_ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">kline_type:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">list&#91;dict&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">&quot;&quot;&quot;Walk backwards from end_ts to start_ts, 500 bars at a time.&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">bars:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict&#91;int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{}<\/span><span style=\"color: #D8DEE9FF\">                  <\/span><span style=\"color: #616E88\"># keyed by bar time -&gt; dedupes overlaps<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">cursor<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">end_ts<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">while<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">cursor<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">start_ts:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">batch<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fetch_klines<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">kline_type,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">500<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">cursor<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        time.sleep<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">REQUEST_INTERVAL<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">not<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">batch:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">break<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">b<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> batch:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;t&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">start_ts<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">end_ts:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #88C0D0\">bars&#91;t&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;pair&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;interval&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">1m<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;ts&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;open&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;o&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;high&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;h&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;low&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;l&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;close&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;c&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">&quot;source&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">rest<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                }<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">oldest<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">min<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">int(b&#91;<\/span><span style=\"color: #88C0D0\">&quot;t&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">b<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">in<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">batch<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">oldest<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">cursor:<\/span><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #616E88\"># no progress: stop instead of looping forever<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">break<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">cursor<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">oldest<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">-<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># save_checkpoint(pair, cursor)  -&gt; resume from here after a crash<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> [bars&#91;t&#93; <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> sorted<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">bars<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">]      <\/span><span style=\"color: #616E88\"># API returns newest-first; store oldest-first<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">A few decisions here are worth copying regardless of provider:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Prices go into <code>Decimal<\/code>, not <code>float<\/code>.<\/strong> A float <code>1.08425<\/code> is not exactly <code>1.08425<\/code>, and the error compounds across a million bars of P&amp;L arithmetic.<\/li>\n\n\n\n<li><strong>Raw responses are written to disk before parsing.<\/strong> When a backtest result looks wrong three weeks from now, you&#8217;ll want to know whether the problem is in the data or in your normalization code.<\/li>\n\n\n\n<li><strong>Bars are keyed by timestamp.<\/strong> Overlapping pages, retried requests, and reruns produce the same result. This idempotency is what makes gap repair safe later.<\/li>\n\n\n\n<li><strong>The <code>oldest >= cursor<\/code> guard<\/strong> stops an infinite loop if the API ever returns the same page twice.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Validate before research touches the data<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Most forex validation code produces false alarms, because it treats the weekend as missing data. The market closes on Friday around 17:00 New York time and reopens on Sunday around 17:00 New York time (21:00 or 22:00 UTC depending on daylight saving). A validator has to know that.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from datetime import datetime, timezone\n\n\ndef fx_market_closed(ts: int) -> bool:\n    \"\"\"Conservative weekend window in UTC. Covers both DST regimes.\"\"\"\n    dt = datetime.fromtimestamp(ts, tz=timezone.utc)\n    wd, hour = dt.weekday(), dt.hour             # Mon=0 ... Sun=6\n    return (wd == 4 and hour >= 21) or wd == 5 or (wd == 6 and hour &lt; 22)\n\n\ndef validate(bars: list&#91;dict&#93;, step: int = 60) -> list&#91;str&#93;:\n    issues = []\n    for prev, cur in zip(bars, bars&#91;1:&#93;):\n        if cur&#91;\"ts\"&#93; &lt;= prev&#91;\"ts\"&#93;:\n            issues.append(f\"order\/duplicate at {cur&#91;'ts'&#93;}\")\n        gap = range(prev&#91;\"ts\"&#93; + step, cur&#91;\"ts\"&#93;, step)\n        missing = &#91;t for t in gap if not fx_market_closed(t)&#93;\n        if len(missing) > 5:                    # tolerate a few quiet minutes\n            issues.append(f\"gap of {len(missing)} open-market minutes after {prev&#91;'ts'&#93;}\")\n\n    for b in bars:\n        if not (b&#91;\"low\"&#93; &lt;= min(b&#91;\"open\"&#93;, b&#91;\"close\"&#93;) &lt;= max(b&#91;\"open\"&#93;, b&#91;\"close\"&#93;) &lt;= b&#91;\"high\"&#93;):\n            issues.append(f\"OHLC inconsistency at {b&#91;'ts'&#93;}\")\n        if b&#91;\"low\"&#93; &lt;= 0:\n            issues.append(f\"non-positive price at {b&#91;'ts'&#93;}\")\n    return issues<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">from<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">datetime<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">datetime,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">timezone<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fx_market_closed<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bool:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">&quot;&quot;&quot;Conservative weekend window in UTC. Covers both DST regimes.&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">dt<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">datetime.fromtimestamp<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">tz=timezone.utc<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">wd,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">hour<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dt.weekday<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dt.hour<\/span><span style=\"color: #D8DEE9FF\">             <\/span><span style=\"color: #616E88\"># Mon=0 ... Sun=6<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> (wd <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">4<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">and<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">hour<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">21<\/span><span style=\"color: #D8DEE9FF\">) or wd == 5 or <\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">wd<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">6<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">and<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">hour<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">22<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">validate<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">bars:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">list&#91;dict&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">step:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">60<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">list&#91;str&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">issues<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> []<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">prev, cur<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> zip<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">bars,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bars&#91;<\/span><span style=\"color: #B48EAD\">1<\/span><span style=\"color: #A3BE8C\">:&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">cur&#91;<\/span><span style=\"color: #88C0D0\">&quot;ts&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">prev&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">ts<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">issues.append(f<\/span><span style=\"color: #88C0D0\">&quot;order\/duplicate at {cur&#91;&#39;ts&#39;&#93;}&quot;<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">gap<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">range<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">prev&#91;<\/span><span style=\"color: #88C0D0\">&quot;ts&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">step,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">cur&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">ts<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">step<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">missing<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> &#91;t <\/span><span style=\"color: #A3BE8C\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">in<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">gap<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">not<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fx_market_closed<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">t<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">len(missing<\/span><span style=\"color: #D8DEE9FF\">) <\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> 5:                    <\/span><span style=\"color: #616E88\"># tolerate a few quiet minutes<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">issues.append(f<\/span><span style=\"color: #88C0D0\">&quot;gap of {len(missing)} open-market minutes after {prev&#91;&#39;ts&#39;&#93;}&quot;<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">b<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> bars:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">not<\/span><span style=\"color: #D8DEE9FF\"> (b&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">low<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #D8DEE9FF\">&#93; <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\">= min<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;open&quot;<\/span><span style=\"color: #88C0D0\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">b&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">close<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\">= max<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;open&quot;<\/span><span style=\"color: #88C0D0\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">b&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">close<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\">= b<\/span><span style=\"color: #ECEFF4\">&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">high<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">&#93;<\/span><span style=\"color: #D8DEE9FF\">):<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">issues.append(f<\/span><span style=\"color: #88C0D0\">&quot;OHLC inconsistency at {b&#91;&#39;ts&#39;&#93;}&quot;<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;low&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">0<\/span><span style=\"color: #A3BE8C\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">issues.append(f<\/span><span style=\"color: #88C0D0\">&quot;non-positive price at {b&#91;&#39;ts&#39;&#93;}&quot;<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">issues<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Also check:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The in-progress bar.<\/strong> The newest bar in a REST response is usually still forming. If your backfill ends at &#8220;now,&#8221; drop the last bar or overwrite it on the next run.<\/li>\n\n\n\n<li><strong>Daily bar boundaries.<\/strong> Forex has no universal &#8220;day.&#8221; Providers close daily bars at UTC midnight, 17:00 New York, or something else. Rebuild a few daily bars from 1-minute data and compare before you use daily candles for anything session-sensitive.<\/li>\n\n\n\n<li><strong>Holidays and thin sessions.<\/strong> Christmas and New Year&#8217;s produce real, legitimate gaps and wide spreads. Mark them; don&#8217;t interpolate them.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">The backtesting pitfall that matters most: spread<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/docs.infoway.io\/en-docs\/rest-api\/market-data\/post-candlestick-real-historical\" target=\"_blank\" rel=\"noopener\">Candles<\/a> are built from a single price series. They don&#8217;t tell you what it would have cost to cross the spread at each moment. A backtest that buys and sells at the candle price is assuming a zero-cost market that doesn&#8217;t exist.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s not a rounding error. Take a strategy that makes 10 round trips a day on EURUSD with an average all-in cost of 0.8 pips per round trip. That&#8217;s 8 pips a day, or <strong>about 2,000 pips a year<\/strong>. Plenty of intraday strategies that look profitable on mid prices lose money once you subtract that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Spread also isn&#8217;t constant. It&#8217;s tightest when London and New York overlap, widens through the Asian session for EUR and GBP pairs, and blows out around the 17:00 New York rollover and major data releases. A fixed spread assumption understates costs exactly where many strategies trade most.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The practical fix: <strong>record bid\/ask live, build a spread profile, and apply it in the backtest.<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from collections import defaultdict\nfrom statistics import median\n\n# Built from live depth snapshots (see Part 2): spread per pair, per hour-of-week\nspread_samples: dict[tuple&#91;str, int&#93;, list&#91;Decimal&#93;] = defaultdict(list)\n\n\ndef record_spread(pair: str, ts_ms: int, bid: Decimal, ask: Decimal) -> None:\n    dt = datetime.fromtimestamp(ts_ms \/ 1000, tz=timezone.utc)\n    spread_samples&#91;(pair, dt.weekday() * 24 + dt.hour)&#93;.append(ask - bid)\n\n\ndef expected_spread(pair: str, ts: int, fallback: Decimal) -> Decimal:\n    dt = datetime.fromtimestamp(ts, tz=timezone.utc)\n    samples = spread_samples.get((pair, dt.weekday() * 24 + dt.hour))\n    return median(samples) if samples else fallback\n\n\ndef fill_price(side: str, ref_price: Decimal, spread: Decimal) -> Decimal:\n    \"\"\"Buys pay the ask, sells hit the bid.\"\"\"\n    half = spread \/ 2\n    return ref_price + half if side == \"buy\" else ref_price - half<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">from<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">collections<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">defaultdict<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">from<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">statistics<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">median<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Built from live depth snapshots (see Part 2): spread per pair, per hour-of-week<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">spread_samples:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict[tuple&#91;str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">list&#91;Decimal&#93;]<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">defaultdict<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">list<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">record_spread<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ts_ms:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">dt<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">datetime.fromtimestamp<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts_ms<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">\/<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1000<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">tz=timezone.utc<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">spread_samples&#91;(pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dt.weekday<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">*<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">24<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dt.hour<\/span><span style=\"color: #D8DEE9FF\">)&#93;.append<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ask<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">-<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">expected_spread<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fallback:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">dt<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">datetime.fromtimestamp<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">tz=timezone.utc<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">samples<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">spread_samples.get<\/span><span style=\"color: #ECEFF4\">((<\/span><span style=\"color: #D8DEE9FF\">pair<\/span><span style=\"color: #81A1C1\">,<\/span><span style=\"color: #D8DEE9FF\"> dt.weekday() <\/span><span style=\"color: #81A1C1\">*<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">24<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">+<\/span><span style=\"color: #D8DEE9FF\"> dt.hour<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">median<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">samples<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">samples<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">else<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fallback<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">fill_price<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">side:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ref_price:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">spread:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">&quot;&quot;&quot;Buys pay the ask, sells hit the bid.&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">half<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">spread<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">\/<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">2<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ref_price<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">half<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">side<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">buy<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">else<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ref_price<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">-<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">half<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Two more rules that prevent the most common self-inflicted backtest bugs:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Never fill on the bar that generated the signal.<\/strong> If your signal uses a bar&#8217;s close, the earliest realistic fill is the <em>next<\/em> bar&#8217;s open. Filling at the same close is lookahead bias.<\/li>\n\n\n\n<li><strong>Be careful when resampling.<\/strong> Building 15-minute bars from 1-minute bars is fine for prices, but the spread at the bar close is not the average spread over the bar. Keep spread on its own time series.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Part 2: WebSocket for Real-Time Forex Streaming<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to subscribe to<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">On Infoway, forex streams over:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"> <code>wss:\/\/data.infoway.io\/ws?business=common&amp;apikey=...<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">with three subscription types:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Request code<\/th><th>Data<\/th><th>Use it for<\/th><\/tr><\/thead><tbody><tr><td><code>10003<\/code><\/td><td>Depth (bid\/ask levels)<\/td><td>Live spread, quote-driven signals, the spread profile above<\/td><\/tr><tr><td><code>10006<\/code><\/td><td>Candles (1-min to 1-year)<\/td><td>Live bars that match your backtest&#8217;s bar format<\/td><\/tr><tr><td><code>10000<\/code><\/td><td>Trades<\/td><td>Last-price displays, tick-level event streams<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Subscribing to both <strong>depth and 1-minute candles<\/strong> gives you the two things a live strategy needs: the same bar format your backtest used, and the real bid\/ask you&#8217;ll actually trade against.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The rules that will disconnect you if you ignore them<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Heartbeat:<\/strong> send a <code>10010<\/code> message regularly. A connection with no heartbeat for 1 minute is closed (error <code>513<\/code>). Every 30 seconds gives you margin.<\/li>\n\n\n\n<li><strong>Request budget:<\/strong> subscribe, unsubscribe, and heartbeat messages share a limit of <strong>60 per minute per connection<\/strong>. Subscribe to all pairs in <strong>one message<\/strong> (comma-separated codes), not one message per pair, or a reconnect with a large watchlist can trip the limit and get disconnected again in a loop.<\/li>\n\n\n\n<li><strong>Connection count:<\/strong> plans allow a fixed number of connections (code <code>427<\/code> when exceeded). Make sure there&#8217;s exactly one reconnect loop. Duplicate reconnect tasks are a classic way to burn through it.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">A resilient consumer with automatic gap repair<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A reconnect restores the <em>stream<\/em>, not the <em>data<\/em>. Whatever happened while you were disconnected is gone from the WebSocket&#8217;s point of view, so the consumer must track the last bar it accepted and backfill the gap over REST after reconnecting.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>import asyncio\nimport random\nimport uuid\n\nimport websockets\n\nWS_URL = f\"wss:\/\/data.infoway.io\/ws?business=common&amp;apikey={API_KEY}\"\nPAIRS = &#91;\"EURUSD\", \"GBPUSD\", \"USDJPY\", \"AUDUSD\"&#93;\n\n\ndef msg(code: int, data: dict | None = None) -> str:\n    m = {\"code\": code, \"trace\": uuid.uuid4().hex}\n    if data is not None:\n        m&#91;\"data\"&#93; = data\n    return json.dumps(m)\n\n\nclass ForexStream:\n    def __init__(self, store):\n        self.store = store                         # anything with upsert_bar() \/ write_quote()\n        self.last_bar_ts: dict&#91;str, int&#93; = {}      # last *completed* 1-min bar per pair\n        self.open_bar: dict&#91;str, dict&#93; = {}        # bar currently forming\n\n    async def run(self) -> None:\n        delay = 5\n        while True:                                # the ONLY reconnect loop\n            try:\n                async with websockets.connect(WS_URL, ping_interval=None) as ws:\n                    codes = \",\".join(PAIRS)\n                    await ws.send(msg(10003, {\"codes\": codes}))                         # depth\n                    await ws.send(msg(10006, {\"arr\": &#91;{\"type\": 1, \"codes\": codes}&#93;}))  # 1-min bars\n                    delay = 5\n                    hb = asyncio.create_task(self._heartbeat(ws))\n                    if self.last_bar_ts:                       # this is a reconnect\n                        asyncio.create_task(self._repair_gaps())\n                    try:\n                        async for raw in ws:\n                            self._dispatch(json.loads(raw))\n                    finally:\n                        hb.cancel()\n            except (websockets.ConnectionClosed, OSError) as e:\n                print(f\"disconnected: {e!r}\")\n            await asyncio.sleep(delay + random.uniform(0, delay \/ 2))   # jittered backoff\n            delay = min(delay * 2, 60)\n\n    async def _heartbeat(self, ws) -> None:\n        while True:\n            await asyncio.sleep(30)\n            await ws.send(msg(10010))\n\n    def _dispatch(self, m: dict) -> None:\n        code, data = m.get(\"code\"), m.get(\"data\")\n        if code == 10005:                          # depth -> top of book\n            bid, ask = Decimal(data&#91;\"b\"&#93;&#91;0&#93;&#91;0&#93;), Decimal(data&#91;\"a\"&#93;&#91;0&#93;&#91;0&#93;)\n            self.store.write_quote(normalize_quote(data&#91;\"s\"&#93;, bid, ask, data&#91;\"t\"&#93;, \"websocket\"))\n            record_spread(data&#91;\"s\"&#93;, data&#91;\"t\"&#93;, bid, ask)\n        elif code == 10008:                        # candle update\n            self._on_bar(data)\n        elif code not in (200, 10001, 10004, 10007, 10010, 10011):\n            print(f\"unexpected message: {m}\")       # surface errors, don't swallow them\n\n    def _on_bar(self, d: dict) -> None:\n        pair, t = d&#91;\"s\"&#93;, int(d&#91;\"t\"&#93;)\n        prev = self.open_bar.get(pair)\n        # The current bar is pushed repeatedly as it updates.\n        # It's final only once a bar with a later timestamp arrives.\n        if prev and t > prev&#91;\"ts\"&#93;:\n            self.store.upsert_bar(prev)\n            self.last_bar_ts&#91;pair&#93; = prev&#91;\"ts\"&#93;\n        self.open_bar&#91;pair&#93; = normalize_bar(pair, t, d, \"websocket\")\n\n    async def _repair_gaps(self) -> None:\n        now = int(time.time())\n        for pair, last_ts in self.last_bar_ts.items():\n            missing = (now - last_ts) \/\/ 60 + 2      # small overlap on purpose\n            if missing &lt;= 2:\n                continue\n            bars = await asyncio.to_thread(fetch_klines, pair, 1, min(missing, 500))\n            for b in bars:\n                t = int(b&#91;\"t\"&#93;)\n                if last_ts &lt; t &lt; now \/\/ 60 * 60:     # completed bars only\n                    self.store.upsert_bar(normalize_bar(pair, t, b, \"rest\"))\n            await asyncio.sleep(REQUEST_INTERVAL)<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">random<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">uuid<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">import<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">websockets<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">WS_URL<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">f<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">wss:\/\/data.infoway.io\/ws?business=common&amp;apikey={API_KEY}<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">PAIRS<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> &#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">EURUSD<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #D8DEE9FF\">, <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">GBPUSD<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">USDJPY<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">AUDUSD<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">msg<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">code:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">|<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">None<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">m<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">code<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">code,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">trace<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">uuid.uuid4<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #A3BE8C\">.hex}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">data<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">is<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">not<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">m&#91;<\/span><span style=\"color: #88C0D0\">&quot;data&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">json.dumps<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">m<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">class<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ForexStream:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">__init__<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">store<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">self.store<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">store<\/span><span style=\"color: #D8DEE9FF\">                         <\/span><span style=\"color: #616E88\"># anything with upsert_bar() \/ write_quote()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">self.last_bar_ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict&#91;str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{}<\/span><span style=\"color: #D8DEE9FF\">      <\/span><span style=\"color: #616E88\"># last *completed* 1-min bar per pair<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">self.open_bar:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict&#91;str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{}<\/span><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># bar currently forming<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">async<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">run<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">5<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">while<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">True:<\/span><span style=\"color: #D8DEE9FF\">                                <\/span><span style=\"color: #616E88\"># the ONLY reconnect loop<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">try:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #88C0D0\">async<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">websockets.connect<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">WS_URL,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ping_interval=None<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">as<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">codes<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">.join<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">PAIRS<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws.send<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">msg(10003,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">codes<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">codes}<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">)                         <\/span><span style=\"color: #616E88\"># depth<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws.send<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">msg(10006,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">arr<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> &#91;{<\/span><span style=\"color: #88C0D0\">&quot;type&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">codes<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">codes}&#93;}<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">)  <\/span><span style=\"color: #616E88\"># 1-min bars<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">5<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">hb<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio.create_task<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self._heartbeat(ws<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">self.last_bar_ts:<\/span><span style=\"color: #D8DEE9FF\">                       <\/span><span style=\"color: #616E88\"># this is a reconnect<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                        <\/span><span style=\"color: #88C0D0\">asyncio.create_task(self._repair_gaps(<\/span><span style=\"color: #D8DEE9FF\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">try:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                        <\/span><span style=\"color: #88C0D0\">async<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">raw<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">in<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                            <\/span><span style=\"color: #88C0D0\">self._dispatch(json.loads(raw<\/span><span style=\"color: #D8DEE9FF\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">finally:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                        <\/span><span style=\"color: #88C0D0\">hb.cancel<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            except <\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">websockets.ConnectionClosed,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">OSError<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> as e:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #88C0D0\">print<\/span><span style=\"color: #D8DEE9FF\">(f<\/span><span style=\"color: #88C0D0\">&quot;disconnected: {e!r}&quot;<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio.sleep<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">random.uniform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">0,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">\/<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">2<\/span><span style=\"color: #ECEFF4\">))<\/span><span style=\"color: #D8DEE9FF\">   <\/span><span style=\"color: #616E88\"># jittered backoff<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">min<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">delay<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">*<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">2<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">60<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">async<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">_heartbeat<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">while<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">True:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio.sleep<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">30<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ws.send<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">msg(10010<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">_dispatch<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">m:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">code,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">m.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;code&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">m.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">&quot;data&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">code<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10005<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\">                          <\/span><span style=\"color: #616E88\"># depth -&gt; top of book<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">bid,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">data&#91;<\/span><span style=\"color: #88C0D0\">&quot;b&quot;<\/span><span style=\"color: #88C0D0\">&#93;&#91;0&#93;&#91;0&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">data&#91;<\/span><span style=\"color: #88C0D0\">&quot;a&quot;<\/span><span style=\"color: #88C0D0\">&#93;&#91;0&#93;&#91;0&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">self.store.write_quote(normalize_quote(data&#91;<\/span><span style=\"color: #88C0D0\">&quot;s&quot;<\/span><span style=\"color: #88C0D0\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">websocket<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #D8DEE9FF\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">record_spread(data&#91;<\/span><span style=\"color: #88C0D0\">&quot;s&quot;<\/span><span style=\"color: #88C0D0\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">data&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">elif<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">code<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10008<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\">                        <\/span><span style=\"color: #616E88\"># candle update<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">self._on_bar(data<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">elif<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">code<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">not<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">in<\/span><span style=\"color: #D8DEE9FF\"> (200, <\/span><span style=\"color: #B48EAD\">10001<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10004<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10007<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10010<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">10011<\/span><span style=\"color: #D8DEE9FF\">):<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">print<\/span><span style=\"color: #D8DEE9FF\">(f<\/span><span style=\"color: #88C0D0\">&quot;unexpected message: {m}&quot;<\/span><span style=\"color: #D8DEE9FF\">)       <\/span><span style=\"color: #616E88\"># surface errors, don&#39;t swallow them<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">_on_bar<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">d:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">d&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">s<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">d&#91;<\/span><span style=\"color: #88C0D0\">&quot;t&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">prev<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">self.open_bar.get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># The current bar is pushed repeatedly as it updates.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># It&#39;s final only once a bar with a later timestamp arrives.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">prev<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">and<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">prev&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">ts<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">self.store.upsert_bar(prev<\/span><span style=\"color: #D8DEE9FF\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">self.last_bar_ts&#91;pair&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">prev&#91;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">ts<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">self.open_bar&#91;pair&#93;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">normalize_bar<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">d,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">websocket<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">async<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">_repair_gaps<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">self<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">None:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">now<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">time.time<\/span><span style=\"color: #ECEFF4\">())<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">pair, last_ts<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">self.last_bar_ts.items<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">missing<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> (now <\/span><span style=\"color: #A3BE8C\">-<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">last_ts<\/span><span style=\"color: #D8DEE9FF\">) \/\/ 60 + 2      <\/span><span style=\"color: #616E88\"># small overlap on purpose<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">missing<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">2<\/span><span style=\"color: #A3BE8C\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #81A1C1\">continue<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">bars<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio.to_thread<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">fetch_klines,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">1<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">min<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">missing,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">500<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #81A1C1\">for<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">b<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">in<\/span><span style=\"color: #D8DEE9FF\"> bars:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #88C0D0\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">b&#91;<\/span><span style=\"color: #88C0D0\">&quot;t&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">last_ts<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">&lt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">now<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">\/\/<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">60<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">*<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">60<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\">     <\/span><span style=\"color: #616E88\"># completed bars only<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">                    <\/span><span style=\"color: #88C0D0\">self.store.upsert_bar(normalize_bar(pair,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">t,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">b,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">rest<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #D8DEE9FF\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">await<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">asyncio.sleep<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">REQUEST_INTERVAL<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Why this works:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Reconnect first, repair second.<\/strong> The stream comes back before the REST backfill starts, so no second gap opens while you&#8217;re repairing the first.<\/li>\n\n\n\n<li><strong>The overlap is deliberate.<\/strong> Repair requests a couple of extra bars, and because bars are upserted by <code>(pair, interval, ts)<\/code>, overlapping REST and WebSocket bars collapse into one record. Without idempotent writes, every repair creates duplicates.<\/li>\n\n\n\n<li><strong>Only completed bars are written.<\/strong> The forming bar is left to the live stream.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Be clear about what can and can&#8217;t be recovered. Missing <strong>1-minute bars<\/strong> can be fully restored over REST. Missing <strong>bid\/ask snapshots<\/strong> can&#8217;t, because there&#8217;s no historical depth endpoint to replay them from. Log those windows as known gaps in the quote series rather than pretending they&#8217;re complete. For long outages (more than 500 minutes), reuse the paged <code>backfill()<\/code> from Part 1 instead of a single request.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">One Canonical Data Model for Both Paths<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Your strategy code should never know whether a bar came from REST or WebSocket. The two sources disagree on almost everything at the format level:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th><\/th><th>REST candles<\/th><th>WebSocket candles<\/th><th>WebSocket depth<\/th><\/tr><\/thead><tbody><tr><td>Timestamp unit<\/td><td>seconds (string)<\/td><td>seconds (integer)<\/td><td>milliseconds<\/td><\/tr><tr><td>Prices<\/td><td>strings<\/td><td>strings<\/td><td>strings, nested arrays<\/td><\/tr><tr><td>Ordering<\/td><td>newest first<\/td><td>push order<\/td><td>push order<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">So everything passes through one normalization layer:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>def normalize_bar(pair: str, ts: int, raw: dict, source: str) -> dict:\n    return {\n        \"pair\": pair,                       # symbol, e.g. \"EURUSD\"\n        \"base\": pair&#91;:3&#93;, \"quote\": pair&#91;3:&#93;,\n        \"interval\": \"1m\",\n        \"ts\": int(ts),                      # bar open time, UTC seconds\n        \"open\": Decimal(raw&#91;\"o\"&#93;), \"high\": Decimal(raw&#91;\"h\"&#93;),\n        \"low\": Decimal(raw&#91;\"l\"&#93;),  \"close\": Decimal(raw&#91;\"c\"&#93;),\n        \"source\": source,                   # \"rest\" | \"websocket\"\n        \"received_at\": time.time(),\n    }\n\n\ndef normalize_quote(pair: str, bid: Decimal, ask: Decimal, ts_ms: int, source: str) -> dict:\n    return {\n        \"pair\": pair,\n        \"base\": pair&#91;:3&#93;, \"quote\": pair&#91;3:&#93;,\n        \"bid\": bid, \"ask\": ask,\n        \"mid\": (bid + ask) \/ 2,\n        \"spread\": ask - bid,\n        \"event_ts_ms\": int(ts_ms),          # provider event time\n        \"received_ts_ms\": int(time.time() * 1000),\n        \"source\": source,\n    }<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">normalize_bar<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ts:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">raw:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">source:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;pair&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair,<\/span><span style=\"color: #D8DEE9FF\">                       <\/span><span style=\"color: #616E88\"># symbol, e.g. &quot;EURUSD&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;base&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair&#91;:3&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">quote<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair&#91;<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #A3BE8C\">:&#93;,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;interval&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">1m<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;ts&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\">                      <\/span><span style=\"color: #616E88\"># bar open time, UTC seconds<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;open&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">raw&#91;<\/span><span style=\"color: #88C0D0\">&quot;o&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">high<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">raw&#91;<\/span><span style=\"color: #88C0D0\">&quot;h&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;low&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">raw&#91;<\/span><span style=\"color: #88C0D0\">&quot;l&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">close<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">raw&#91;<\/span><span style=\"color: #88C0D0\">&quot;c&quot;<\/span><span style=\"color: #88C0D0\">&#93;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;source&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">source,<\/span><span style=\"color: #D8DEE9FF\">                   <\/span><span style=\"color: #616E88\"># &quot;rest&quot; | &quot;websocket&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;received_at&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">time.time<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #A3BE8C\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    }<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">normalize_quote<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">pair:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">Decimal,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ts_ms:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">source:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">str<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> -<\/span><span style=\"color: #81A1C1\">&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">dict:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;pair&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;base&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair&#91;:3&#93;,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">quote<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">pair&#91;<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #A3BE8C\">:&#93;,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;bid&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">ask<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;mid&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> (bid <\/span><span style=\"color: #A3BE8C\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask<\/span><span style=\"color: #D8DEE9FF\">) \/ 2,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;spread&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">ask<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">-<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">bid,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;event_ts_ms&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">ts_ms<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #A3BE8C\">,<\/span><span style=\"color: #D8DEE9FF\">          <\/span><span style=\"color: #616E88\"># provider event time<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;received_ts_ms&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">int<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">time.time<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> * 1000),<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">&quot;source&quot;<\/span><span style=\"color: #88C0D0\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">source,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    }<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Keeping <strong>both the event time and the receive time<\/strong> costs almost nothing and pays for itself the first time you investigate latency. The difference between the two is your end-to-end lag, and a growing gap between them is often the earliest sign that a consumer is falling behind.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Reference Architecture<\/h2>\n\n\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6ad4211e69693&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6ad4211e69693\" class=\"wp-block-image size-large wp-lightbox-container\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"570\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on--pointerdown=\"actions.preloadImage\" data-wp-on--pointerenter=\"actions.preloadImageWithDelay\" data-wp-on--pointerleave=\"actions.cancelPreload\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-1024x570.png\" alt=\"Production Forex data architecture\" class=\"wp-image-280\" srcset=\"https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-1024x570.png 1024w, https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-300x167.png 300w, https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-768x427.png 768w, https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-1536x854.png 1536w, https:\/\/blog.infoway.io\/en\/wp-content\/uploads\/sites\/2\/2026\/10\/forex-architecture-diagram-2048x1139-1.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\tdata-wp-bind--aria-label=\"state.thisImage.triggerButtonAriaLabel\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.thisImage.buttonRight\"\n\t\t\tdata-wp-style--top=\"state.thisImage.buttonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Monitor these, per pair rather than just per connection:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Freshness:<\/strong> seconds since the last quote. During market hours, a pair that goes quiet while others keep updating means a broken subscription, not a quiet market.<\/li>\n\n\n\n<li><strong>Reconnect count and repair volume:<\/strong> how often you reconnect, and how many bars each repair writes.<\/li>\n\n\n\n<li><strong>Lag:<\/strong> <code>received_ts - event_ts<\/code>, at the 50th and 99th percentile.<\/li>\n\n\n\n<li><strong>Validation failures<\/strong> on each backfill run.<\/li>\n\n\n\n<li><strong>REST <code>429<\/code>s:<\/strong> if they show up regularly, your request interval is wrong.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">A One-Day Proof of Concept<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before committing to any forex data provider, run this test. It surfaces most integration problems in a day:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Backfill one month<\/strong> of 1-minute EURUSD over REST. Run the validator. Expect zero open-market gaps larger than a few minutes.<\/li>\n\n\n\n<li><strong>Stream the same pair<\/strong> over WebSocket for a few hours, recording depth and 1-minute bars.<\/li>\n\n\n\n<li><strong>Compare overlapping bars.<\/strong> Bars from REST and WebSocket for the same minutes should match after normalization. If they don&#8217;t, find out why before you build anything else.<\/li>\n\n\n\n<li><strong>Kill the connection<\/strong> (pull the network, or kill the process) for 10 minutes during market hours. Restart. Confirm the repair job restores every missing bar and writes no duplicates.<\/li>\n\n\n\n<li><strong>Inspect the spread profile.<\/strong> Spreads should clearly widen around 17:00 New York time and tighten during the London\/New York overlap. If spread looks flat all day, you&#8217;re not looking at real bid\/ask data.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pass criteria:<\/strong> no unexplained gaps, identical schemas from both paths, repair completes within a minute of reconnecting, and duplicate count after repair is zero.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Things to verify with any provider while you&#8217;re at it: pair coverage, historical depth per interval, whether bid\/ask is available (live and historical), timestamp semantics, the daily bar boundary, rate and connection limits, and whether commercial use is covered by your plan.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Which Interface for Which Job<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Task<\/th><th>Interface<\/th><th>Notes<\/th><\/tr><\/thead><tbody><tr><td>Backtesting and research datasets<\/td><td>REST<\/td><td>Paged, checkpointed, validated<\/td><\/tr><tr><td>Scheduled ingestion (e.g. nightly)<\/td><td>REST<\/td><td>Idempotent upserts so reruns are safe<\/td><\/tr><tr><td>Gap repair after an outage<\/td><td>REST<\/td><td>Bars recoverable; bid\/ask snapshots are not<\/td><\/tr><tr><td>Live dashboards and alerts<\/td><td>WebSocket<\/td><td>One connection, batched subscriptions<\/td><\/tr><tr><td>Live spread and quote tracking<\/td><td>WebSocket (depth)<\/td><td>Also feeds the backtest spread model<\/td><\/tr><tr><td>Signal engines on live bars<\/td><td>WebSocket (candles)<\/td><td>Same bar format as the backtest<\/td><\/tr><tr><td>Production trading systems<\/td><td>Both<\/td><td>One schema, tested recovery path<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The architecture comes down to three rules. Use REST for anything bounded and repeatable. Use WebSocket for anything continuous. Connect the two with a shared data model and a gap-repair path you have actually tested by pulling the plug.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Try It With Infoway API<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Infoway API provides forex, precious metals, energy, stocks, and crypto through the same REST and WebSocket interfaces, so the architecture above extends to multi-asset systems without a second integration. New accounts get a <strong>7-day free trial<\/strong> with no credit card required, which is enough time to run the proof of concept end to end.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/infoway.io\/en\/create-account\" target=\"_blank\" rel=\"noopener\">Create a free account<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.infoway.io\/en-docs\/readme\/forex-market-data-api\" target=\"_blank\" rel=\"noopener\">API documentation<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Most forex data problems in production don&#8217;t come from bad data. They come from using the right data through the wrong interface.<\/p>\n","protected":false},"author":1,"featured_media":283,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[8],"tags":[],"class_list":["post-278","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to"],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/posts\/278","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/comments?post=278"}],"version-history":[{"count":3,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/posts\/278\/revisions"}],"predecessor-version":[{"id":282,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/posts\/278\/revisions\/282"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/media\/283"}],"wp:attachment":[{"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/media?parent=278"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/categories?post=278"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.infoway.io\/en\/wp-json\/wp\/v2\/tags?post=278"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}