Linode Thrashing Swap

I got a message from Linode that my vps had been averaging above 90% cpu usage over two hours. By the time I had checked my email and looked at the graphs on the web linode manager, the problem had been going on for 12 hours — cpu hovering between 90 and 100% for that amount of time.

I logged into my server and ran top. A process called kswapd0 was the obvious culprit. That's the kernel swap daemon. Okay, what does one do about a machine that is thrashing swap?

The first advice that I found on the web was to tell Linux to drop caches by putting a value of 1 or 3 in /proc/sys/vm/drop_caches.

# cat /proc/sys/vm/drop_caches
# echo 3 > /proc/sys/vm/drop_caches

I did this and watched the top output, and nothing changed. I tried putting 0 back into that file, but received an error:

# echo 0 > /proc/sys/vm/drop_caches
bash: echo: write error: Invalid argument

Not sure what's going on there, yet.

I asked for advice in the Linode irc channel and among other things, was directed to this helpful article: Troubleshooting Memory and Networking Issues. To see the highest memory consumer on the system:

# ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less

In my case, it was apache2. I restarted apache2 and the swap thrashing ended.