Redis Issue with Prod and Code

[Zack-Rider]

Redis Production Issue — Root Cause & Fix

Date: March 17, 2026
Symptom: All production requests taking 5–10 seconds (local was ~1 second)

---

What Was Happening

The Code Bug

The get function in server/src/utils/redis.ts had two competing timeout mechanisms running simultaneously:

t=0s   → Redis GET command issued
t=2s   → Promise.race timer fires → returns null, code moves on
         BUT the underlying Redis promise is still alive (orphaned)
t=3s   → ioredis commandTimeout fires → rejects the orphaned promise
         → Unhandled promise rejection
         → ioredis emits this as a connection-level 'error' event
         → "Redis connection error" logged
         → Triggers reconnect: connection closed → null → new connection created
t=3s+  → Reconnect loop begins, next requests queue up
         → More timeouts, more reconnects — death spiral

Every request was paying a mandatory 2-second wait for Redis before falling through to the database, making every request 5–10 seconds.

The Infrastructure Issue

On March 15, 2026 at 18:00 UTC+5:30, a modification was applied to the ElastiCache cluster which caused a full restart:

• Redis cache was completely wiped (memory usage dropped from 1.76% → 0.09%)
• Memory Fragmentation Ratio spiked from ~1.62 → 2.73 (normal is 1.0–1.5)
• Cold cache meant every Redis GET was a miss, falling through to the database

Additionally, a cross-AZ mismatch was found:

• ECS Fargate task runs in us-west-1c
• ElastiCache Redis node is in us-west-1b

---

Code Changes Made

File: server/src/utils/redis.ts

1. Removed Promise.race wrapper from get()

The Promise.race was the primary bug — it orphaned the Redis command promise, causing unhandled rejections that ioredis surfaced as connection errors, triggering reconnect storms.

Before:

const get = async (ctx, key) => {
  const timeoutPromise = new Promise<null>((resolve) => {
    setTimeout(() => {
      console.warn(`Redis GET slow for key: ${key}`);
      resolve(null);
    }, 2000);
  });

  const result = await Promise.race([client.get(key), timeoutPromise]);
  return result;
};

After:

const get = async (ctx, key) => {
  try {
    const client = getRedisClient(ctx);
    return await client.get(key);
  } catch (error) {
    console.error('Redis GET error:', error);
    return null;
  }
};

2. Reduced commandTimeout from 3000ms → 200ms

With commandTimeout: 200, if Redis doesn't respond within 200ms, the command fails immediately and the request falls through to the database — instead of blocking for 3 seconds.

Setting	Before	After
commandTimeout	3000ms	200ms
maxRetriesPerRequest	2	1
connectTimeout	10000ms	5000ms

---

Infrastructure Actions Required

1. Fix Memory Fragmentation (do now)

In ElastiCache → Configurations → Parameter groups, edit the parameter group attached to the cluster and set:

activedefrag = yes
active-defrag-enabled = yes

This runs online defragmentation passively — no restart needed. The fragmentation ratio will drop from 2.73 back to ~1.0–1.5 over time.

2. Fix Cross-AZ Mismatch (do soon)

ECS app is in us-west-1c, Redis is in us-west-1b. Options:

• Option A: Add a Redis replica node in us-west-1c and point REDIS_HOST to it
• Option B: Update ECS service subnet to one in us-west-1b

Cross-AZ adds latency and incurs AWS data transfer costs on every Redis command.

---

Expected Outcome After Deploy

Scenario	Before Fix	After Fix
Redis hit (cache warm)	~2s+	<50ms
Redis miss (cache cold)	~5–10s	~1s (falls to DB fast)
Redis slow/down	Reconnect storm	Fail fast, serve from DB

Error Logs

Redis GET error: 360 |             // https://github.com/iojs/io.js/pull/1217
361 |             writable = false;
362 |         }
363 |         if (!writable) {
364 |             if (!this.options.enableOfflineQueue) {
365 |                 command.reject(new Error("Stream isn't writeable and enableOfflineQueue options is false"));
                                         ^
error: Stream isn't writeable and enableOfflineQueue options is false
      at sendCommand (/home/piyush/D_Drive/Coyax/node_modules/ioredis/built/Redis.js:365:36)

Redis SET failed (non-blocking): org:coyaxai Stream isn't writeable and enableOfflineQueue options is false
Redis connection error: error: connect ECONNREFUSED 127.0.0.1:6379
   errno: -111,
 syscall: "connect",
    port: 6379,
 address: "127.0.0.1",
    code: "ECONNREFUSED"

      at new ExceptionWithHostPort (internal:shared:42:10)

Redis retry attempt 5
Redis connection closed
Redis connection error: error: connect ECONNREFUSED 127.0.0.1:6379
   errno: -111,
 syscall: "connect",
    port: 6379,
 address: "127.0.0.1",
    code: "ECONNREFUSED"

      at new ExceptionWithHostPort (internal:shared:42:10)

Redis retry attempt 6
Redis max retries reached, giving up
Redis connection closed
Redis connection ended
Req Time Taken 19645ms