Dokku Performance Issues On VPS

Hey 👋,

long time ago I decided I was done with my shared hosting and wanted something new.

I wanted the solution to be able to run on a cheap VPS at DigitalOcean (link with referral).

I promise I plan to write a post about “how to migrate a site”, but I had this in draft first :)

What was my motivation?

As I said, I will go into more details about why I decided to do this and how I did it in a separate post.

For the moment let’s say I was a bit in a rush, because my shared hosting decided to cancel my SSH access (suddenly, without any previous warning) right after increasing their plan.

More money, less features… ❌

Smooth start

Everything was quite easy to do. And suddenly I had a couple of sites migrated there. Awesome.

But then I migrated my bigger site there, and things started to go wrong. And it wasn’t obvious to know why.

Symptoms

Suddenly the sites went unresponsive and the VPS restarted.

It was difficult to notice because it happened what seemed like randomly.

So I got my hands on and started monitoring my vps instance.

Debugging

I saw the CPU usage was high.

After checking with htop I saw that the snap daemon appeared consistently on the top results when sorting by cpu.

I uninstalled snap (since I was not using any snap packages) and… it did nothing. CPU was still high.

Checking the logs and the Docker processes I saw that the MariaDB instance was restarting. And this was what I saw in the errors:

2021-11-23 18:18:05+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
	command was: mysqld --verbose --help --log-bin-index=/tmp/tmp.N2oDjCY875

I played around stopping/starting some of the sites (since I had a few running on Dokku).

What I found was that when I had only a few sites running, everything was stable, but more sites caused instability

I went and checked the system logs.

Maybe I should have started here. But I always find it difficult to locate them and understand which one is from where (dmesg, journactl, /var/log, …).

...
[430425.077411] Out of memory: Killed process 1804946 (mysqld) total-vm:1072652kB, anon-rss:77728kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:352kB oom_score_adj:0
...
[430425.086045] oom_reaper: reaped process 1804946 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
...

OUT OF MEMORY! Nice, now I knew what the issue was.

🧠 Thanks to this process I realised that I didn’t really understand the data in htop. Because there I saw memory sitting at 700M/981M. Which was ok-ish.

Cheap solution?

Identifying that I was hitting a memory problem was a tough thing.

I didn’t want to spend more money, since the original ideas was to keep it around the same money than my original shared hosting.

And when I was about to go and upgrade my VPS node… I checked the logs again and I saw that SWAP was disabled.

I enabled swap following these steps.

Here you have a quick list of the commands I used. But I think you should check the whole guide above.

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Done 🎉

After enabling swap I saw the CPU go flat and everything was looking great. Yay! 🎉

Just to be sure I created also an alert in Digital Ocean when CPU usage went above 70% for more than 5 minutes.

Now I feel safe. And since this process happened months before this post was published I can say that IT WORKED!

Hope you found this useful! Let me know by clicking the 👍 button below (which I made myself)