Migrating 5 million records teaches what no tutorial tells you
Published on August 24, 2026
𝗠𝗶𝗴𝗿𝗮𝘁𝗶𝗻𝗴 𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝗿𝗲𝗰𝗼𝗿𝗱𝘀 𝘁𝗲𝗮𝗰𝗵𝗲𝘀 𝘄𝗵𝗮𝘁 𝗻𝗼 𝘁𝘂𝘁𝗼𝗿𝗶𝗮𝗹 𝘁𝗲𝗹𝗹𝘀 𝘆𝗼𝘂 🔥
Running a migration script without a safety net is like operating without anesthesia. It works... until it doesn't. 😅
I had to rebuild a 𝗠𝗼𝗻𝗴𝗼𝗗𝗕 𝘁𝗶𝗺𝗲 𝘀𝗲𝗿𝗶𝗲𝘀 collection with more than 𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝗿𝗲𝗰𝗼𝗿𝗱𝘀. I ran the process using 𝗡𝗼𝗱𝗲.𝗷𝘀 scripts directly from the server console, with the application in 𝗺𝗮𝗶𝗻𝘁𝗲𝗻𝗮𝗻𝗰𝗲 𝗺𝗼𝗱𝗲. Without losing data. And potentially running multiple times if something failed. That forced me to think differently. 💻
Six principles I learned the hard way 👇
🧪 𝗗𝗿𝘆-𝗿𝘂𝗻
Run the script in simulation mode first — without applying real changes. It estimates how long it takes, catches logic errors, and validates the flow before touching production. With millions of records, this step saves hours of pain.
🗄️ 𝗣𝗿𝗶𝗼𝗿 𝗯𝗮𝗰𝗸𝘂𝗽
Step zero before running any migration. If the script has a logic bug, you need to be able to roll back. At this scale, there's no room for improvising a recovery.
🔁 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆
The script must be able to run N times and always produce the same result. Before inserting or modifying, verify if the target state already exists. Never assume it's the first execution.
🛡️ 𝗥𝗲𝘀𝗶𝗹𝗶𝗲𝗻𝗰𝗲
Save checkpoints: record how far the process got. If it fails halfway, it must resume from there — not start over from scratch. Detailed logging is mandatory, not optional.
⚡ 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆
With 5M records you can't process one by one. Controlled 𝗯𝗮𝘁𝗰𝗵𝗲𝘀 + pauses between batches to avoid saturating the DB in production. The goal: reduce time and resource consumption without breaking anything.
✅ 𝗣𝗼𝘀𝘁-𝗺𝗶𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝘃𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻
The script can finish without errors and still have migrated incorrectly. Verify record counts, integrity, and consistency between source and destination. Without validation, you don't know if it really worked.
A migration script is not code you run once and forget. It's infrastructure. It deserves the same rigor as any production service. 🚀
Have you ever had to migrate data in production under pressure? What would have helped you know beforehand? 👇
Keep flying, Champions! ✈️