MySQL Not Starting After Disk Full: InnoDB Recovery Without a Backup

Many sysadmins have dealt with MySQL not starting after disk full on a Linux server. It is one of the most stressful situations in server management. The service fails silently. Standard restart commands do nothing. However, in most cases you can recover fully — without restoring from a backup.

What Actually Goes Wrong

When the disk fills up mid-write, InnoDB cannot finish flushing its doublewrite buffer. This buffer lives inside ibdata1. It acts as a safety checkpoint. InnoDB writes pages here first, then moves them to their final location on disk.

If that process gets interrupted, page space=0, page number=5 becomes corrupted. That page is the InnoDB transaction system page. Because of this, MySQL aborts startup every single time — even after you free disk space.

The error in /var/log/mysql/error.log will show:

[ERROR] [MY-011906] [InnoDB] Database page corruption on disk or a failed
file read of page [page id: space=0, page number=5].
You may have to recover from a backup.

Despite that message, a backup is rarely needed. The MySQL InnoDB force recovery documentation describes a startup mode that lets you bypass the corruption and dump data safely.

Step 1 — Free Up Disk Space First

MySQL will not start at all if the disk is still full. Check this first:

df -h

You need at least 2–3 GB free. On cPanel/WHM servers, these log files are usually the biggest culprits. Truncate them using > instead of rm. Running services hold file handles open, so deleting the file does not actually free the space:

> /usr/local/cpanel/logs/cpdavd_session_log
> /usr/local/cpanel/logs/error_log
> /usr/local/cpanel/logs/queueprocd.log
journalctl --vacuum-size=200M

Step 2 — Read the Full Error Log

Once disk space is clear, read the error log carefully:

tail -80 /var/log/mysql/error.log

Look for this specific crash trace:

Double_write::init_v1 → dblwr::v1::init → srv_start → CRASH

That trace confirms doublewrite buffer corruption. It also reveals something important. Standard innodb_force_recovery alone will not solve this. You also need to disable the doublewrite buffer. That is the step most recovery guides miss.

Step 3 — Enable Force Recovery in my.cnf

First, find which config file MySQL uses on your system:

mysql --help | grep "Default options" -A2

Then open the config — usually /etc/my.cnf — and add these two lines under [mysqld]:

[mysqld]
innodb_force_recovery = 1
innodb_doublewrite = 0

The second line is the critical one. It tells InnoDB to skip the corrupted buffer on startup. Without it, MySQL keeps crashing even with force_recovery set. After saving, start the service:

systemctl start mysql && systemctl status mysql

If MySQL starts, move to Step 4 immediately. If it still fails, however, increase the recovery level one step at a time up to a maximum of 6:

Important: Never serve live traffic with innodb_force_recovery above 0. Use this mode only to extract your data. Writes in this state can cause further damage.

Step 4 — Dump All Databases Right Away

The moment MySQL starts in recovery mode, dump everything. Do not run queries. Do not restart. Just dump immediately:

mysqldump --all-databases --single-transaction --quick \
  --routines --triggers -u root \
  > /home/alldb_backup_$(date +%F).sql

After it finishes, verify the dump completed cleanly:

tail -5 /home/alldb_backup_$(date +%F).sql
# Good: -- Dump completed on YYYY-MM-DD HH:MM:SS

Do not continue to Step 5 until you confirm that last line. A truncated dump means the data is incomplete.

Step 5 — Rebuild a Clean InnoDB Tablespace

Once you verify the dump, stop MySQL. Then remove the corrupted system files. MySQL will recreate them cleanly on the next start:

systemctl stop mysql
mkdir -p /root/mysql_innodb_backup
cp /var/lib/mysql/ibdata1 /root/mysql_innodb_backup/
cp /var/lib/mysql/#ib_16384_*.dblwr /root/mysql_innodb_backup/
cp -r /var/lib/mysql/#innodb_redo /root/mysql_innodb_backup/ 2>/dev/null
rm /var/lib/mysql/ibdata1
rm -f /var/lib/mysql/ib_logfile*
rm -f /var/lib/mysql/#ib_16384_*.dblwr
rm -rf /var/lib/mysql/#innodb_redo

Next, remove the two recovery lines from /etc/my.cnf. Then start MySQL fresh and restore:

systemctl start mysql
mysql -e "SHOW VARIABLES LIKE 'innodb_force_recovery';"
mysql -u root < /home/alldb_backup_$(date +%F).sql
echo "Restore complete."

Step 6 — Set Up Disk Monitoring

The root cause here was a disk that filled without any warning. On cPanel servers, configure disk alerts right away. Go to WHM → Server Contacts → Disk Usage Warnings and set thresholds at 80% and 90%. The full setup is in the cPanel WHM system configuration documentation.

Also add a cron monitor as a backup. It checks every 30 minutes and emails root when disk use crosses 85%:

echo '*/30 * * * * root DISK=$(df / | awk 'NR==2{print $5}' | tr -d "%"); [ "$DISK" -gt 85 ] && echo "Disk ${DISK}%" | mail -s "DISK WARNING" root' > /etc/cron.d/disk-monitor

Conclusion

The problem of MySQL not starting after disk full is recoverable in most cases without touching a backup. The two-flag fix works like this: innodb_force_recovery = 1 puts MySQL in a readable state. Then innodb_doublewrite = 0 bypasses the corrupted buffer blocking startup. Together, they let you dump your data safely, rebuild a clean tablespace, and restore — usually within an hour. Set up disk monitoring afterward so this never happens again.