The deploy went fine. The infrastructure didn't.
Published on August 17, 2026
𝗧𝗵𝗲 𝗱𝗲𝗽𝗹𝗼𝘆 𝘄𝗲𝗻𝘁 𝗳𝗶𝗻𝗲. 𝗧𝗵𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗱𝗶𝗱𝗻'𝘁.
An hour after the deploy, the instance ran out of RAM. The services that were working perfectly before started failing, one by one. 😅
The new service was concurrent and made HTTP requests. The problem: it 𝗻𝗲𝘃𝗲𝗿 𝗰𝗹𝗼𝘀𝗲𝗱 𝘁𝗵𝗲 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀. Every request opened a new connection and left it there. Concurrency multiplied them until memory collapsed and took down everything sharing the instance. I lived it firsthand at 𝗟𝘂𝗺𝗶𝗻𝗛𝗲𝗮𝗹𝘁𝗵 💻
Dev's fault or DevOps's? 🔥
Honest answer: both.
The 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 didn't manage the connection lifecycle → real bug.
The 𝗗𝗲𝘃𝗢𝗽𝘀 didn't apply memory limits to the service → without a guardrail, one broken service can take down its neighbors.
Production is not just shipping code to the cloud. It's a 𝘀𝗵𝗮𝗿𝗲𝗱 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗰𝗼𝗻𝘁𝗿𝗮𝗰𝘁.
Since that incident, my deploys always include:
✅ 𝗠𝗲𝗺𝗼𝗿𝘆 and CPU limits per service
✅ 𝗛𝗲𝗮𝗹𝘁𝗵 𝗰𝗵𝗲𝗰𝗸𝘀 and restart policies
✅ 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 before the green light
Champion, have you experienced an incident like this? Who took responsibility in your team? 👇
Keep flying, Champions! ✈️