Without a queue in between, a traffic spike takes down your backend

𝗪𝗶𝘁𝗵𝗼𝘂𝘁 𝗮 𝗾𝘂𝗲𝘂𝗲 𝗶𝗻 𝗯𝗲𝘁𝘄𝗲𝗲𝗻, 𝗮 𝘁𝗿𝗮𝗳𝗳𝗶𝗰 𝘀𝗽𝗶𝗸𝗲 𝘁𝗮𝗸𝗲𝘀 𝗱𝗼𝘄𝗻 𝘆𝗼𝘂𝗿 𝗯𝗮𝗰𝗸𝗲𝗻𝗱 𝗯𝗲𝗳𝗼𝗿𝗲 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗿𝗲𝗮𝗰𝘁

If your server processes each job the moment it arrives, the day 500 show up at once, you process 500 failures at the same time 😅

When I set up 𝗻𝟴𝗻 𝘄𝗶𝘁𝗵 𝗮𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗶𝗻𝗴 on 𝗔𝗪𝗦, the problem wasn't scaling instances — the 𝗔𝘂𝘁𝗼 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗚𝗿𝗼𝘂𝗽 already handled that. The real problem was different: if a worker got more executions than it could process, those executions got lost or crashed the process. Scaling machines is pointless if the work gets lost before it reaches them 💻

That's where n8n's 𝗾𝘂𝗲𝘂𝗲 𝗺𝗼𝗱𝗲 came in 👇

📥 The main process doesn't execute anything — it just receives the job and 𝗾𝘂𝗲𝘂𝗲𝘀 it in 𝗥𝗲𝗱𝗶𝘀.

⚙️ Several 𝘄𝗼𝗿𝗸𝗲𝗿𝘀 (one per ASG instance) read from that queue and process when they have free capacity. Nobody gets overloaded because nobody receives more than they can handle.

🔁 If a worker dies mid-execution, the job goes back to the queue and another worker picks it up — with 𝗿𝗲𝘁𝗿𝗶𝗲𝘀 configured instead of losing the work.

📈 Scaling is simple now: if the queue grows, I add more workers. The ASG handles it on its own, but now it's scaling something that actually solves the bottleneck.

The queue doesn't prevent the traffic spike. It absorbs it, and buys you time to scale without losing a single job 🔥

Champion, does your backend process everything the moment it arrives, or does it have something to absorb the hit when it all arrives at once? 🤔

Keep flying, Champions! ✈️