{"id":67184,"date":"2022-09-13T09:01:11","date_gmt":"2022-09-13T09:01:11","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=67184"},"modified":"2022-09-13T09:01:11","modified_gmt":"2022-09-13T09:01:11","slug":"how-i-troubleshoot-swappiness-and-startup-time-on-linux","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=67184","title":{"rendered":"How I troubleshoot swappiness and startup time on Linux"},"content":{"rendered":"

How I troubleshoot swappiness and startup time on Linux<\/span>
\nDavid Both<\/a><\/span>
\nTue, 09\/13\/2022 – 03:00<\/span><\/p>\n

\n

I recently experienced another interesting problem in the Linux startup sequence that has a circumvention\u2013not a solution. It started quite unexpectedly.<\/p>\n

I was writing a couple of articles while making some updates to my personal copy of my series of books, “Using and Administering Linux: Zero to SysAdmin.” I had four instances of LibreOffice Write<\/a> open to doing all that. I had three VMs running with VirtualBox to test some of the things I was writing about. I also had LibreOffice Impress open to work on an unrelated presentation. I like to listen to music, so I had one of several tabs in Firefox open to Pandora, my music streaming service of choice. I had multiple Bash shells open using Konsole with numerous tabs and the Alpine text-mode email client in one. Then there were the various tabs in the Thunar file manager.<\/p>\n

So I had a lot going on. Just like I do now as I write this article.<\/p>\n

The symptoms<\/h2>\n

As I used these open sessions, I noticed that things slowed down considerably while waiting for the system to write a document to the M.3 SSD\u2013a process that should have been really fast. I also noticed that the music was choppy and dropped out completely every few minutes. Overall performance was generally poor. I began to think that Fedora had a serious problem.<\/p>\n

My primary workstation, the one I was working on at the time, has 64GB of RAM and an Intel Core i9 Extreme with 16 cores and Hyperthreading (32 CPUs) that can run as fast as 4.1 GHz using my configured overclocking. So I should not have experienced any slowdowns\u2013or so I thought at the time.<\/p>\n

Determine the problem<\/h2>\n

It did not take long to find the problem because I have experienced similar symptoms before on systems with far less memory. The issue looked like delays due to page swapping. But why?<\/p>\n

I started with one of my go-to tools for problem determination, htop. It showed that the system was using 13.6GB of memory for programs, and most of the rest of the RAM was in cache and buffers. It also showed that swapping was actively occurring and that about 253MB of data was stored in the swap partitions.<\/p>\n

\n
Date &<\/span> Time: 2022<\/span>-08-12<\/span> 10<\/span>:53<\/span>:08
\nUptime: 2<\/span> days, 23<\/span>:47<\/span>:15<\/span>
\nTasks: 200<\/span>, 1559<\/span> thr, 371<\/span> kthr; 4<\/span> running
\nLoad average: 3.97<\/span> 3.05<\/span> 2.08<\/span>
\n\u00a0 \u00a0
\nDisk IO: 202.6<\/span>%<\/span> read: 687M write: 188K
\nNetwork: rx: 0KiB\/<\/span>s tx: 0KiB\/<\/span>s (<\/span>0<\/span>\/<\/span>0<\/span> packets)<\/span>
\nSystemd: running (<\/span>0<\/span>\/<\/span>662<\/span> failed)<\/span> (<\/span>0<\/span>\/<\/span>7912<\/span> jobs<\/span>)<\/span> \u00a0 \u00a0 
\nMem[<\/span>|||||||<\/span>##*@@@@@@@@@@@@@@@@@@@@@@@@@@ \u00a0 \u00a013.6G\/62.5G]<\/span>
\nSwp[<\/span>||<\/span># \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0253M\/18.0G]<\/span><\/div><\/div><\/pre>\n
But that meant I still had lots of memory left the system could use directly for programs and data and more that it could recover from cache and buffers. So why was this system even swapping at all?<\/p>\n
I remembered hearing about the “swappiness” factor in one of my Red Hat training classes<\/a>. But that was a long time ago. I did some searches on “swappiness” to learn about the kernel setting vm.swappiness<\/strong>.<\/p>\n
The default value for this kernel parameter is 60<\/strong>. That represents the percent of free memory not yet in use. When the system reaches that 60% trigger point, it begins to swap, no matter how much free memory is available. My system started swapping when about 0.6 * 62.5GB = 37.5GB of unused memory remained.<\/p>\n
Based on my online reading, I discovered that 10% is a better setting for many Linux systems. With that setting, swapping starts when only 10% of RAM is free.<\/p>\n
I checked the current swappiness setting on my system, and it was set to the default.<\/p>\n
\n
# sysctl vm.swappiness<\/span>
\nvm.swappiness = 60<\/span><\/div><\/div><\/pre>\n
Time to change this kernel setting.<\/p>\n<\/p>\n
\n
More for sysadmins<\/div>\n
\n
Enable Sysadmin blog<\/a><\/div>\n
The Automated Enterprise: A guide to managing IT with automation<\/a><\/div>\n
eBook: Ansible automation for Sysadmins<\/a><\/div>\n
Tales from the field: A system administrator’s guide to IT automation<\/a><\/div>\n
eBook: A guide to Kubernetes for SREs and sysadmins<\/a><\/div>\n
Latest sysadmin articles<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n
Fix the issue<\/h2>\n
I won’t dive into the gory details, but the bottom line is that either of the following commands, run as root, will instantly do the job on a running Linux computer without a reboot.<\/p>\n
\n# <\/span>sysctl -w<\/span> vm.swappiness=10<\/span><\/code><\/span><\/pre>\n
You could also use this next command to do the same thing.<\/p>\n
\n# <\/span>echo<\/span> 10<\/span> ><\/span> \/<\/span>proc\/<\/span>vm\/<\/span>swappiness<\/code><\/span><\/pre>\n
Tecmint has an excellent article about setting kernel parameters.<\/p>\n
Both commands change the live kernel setting in the \/proc<\/code> filesystem. After running either of those commands, you should run the sysctl vm.swappiness<\/code> command to verify that the kernel setting has changed.<\/p>\n
But those commands only change the swappiness value for the currently running system. A reboot returns the value to its default. I needed to ensure that this change is made persistent across reboots.<\/p>\n
But first, the failure<\/h2>\n
To permanently change the kernel vm.swappiness<\/strong> variable, I used the procedure described in my previous article, How I disabled IPv6 on Linux<\/a>, to add the following line to the end of the \/etc\/default\/grub<\/code> file:<\/p>\n
\nGRUB_CMDLINE_LINUX=\"vm.swappiness=1\"<\/code><\/span><\/pre>\n
I then ran the grub2-mkconfig<\/code> command as root to rebuild the \/boot\/grub2\/grub.cfg<\/code> file. However, testing with VMs and real hardware showed that it did not work, and the swappiness value did not change. So I tried another approach.<\/p>\n
And the success<\/h2>\n
Between this failure at startup time, the one I describe in the How I disabled IPv6 on Linux<\/em> article, and other startup issues I explored due to encountering those two, I decided that this was a Linux startup timing problem. In other words, some required services, one of which might be the network itself, were not up and running, which prevented these kernel option changes from being committed to the \/proc<\/code> filesystem, or they were committed and then overwritten when the service started.<\/p>\n
I could make all of these work as they should by adding them to a new file, \/etc\/sysctl.d\/local-sysctl.conf<\/code> with the following content, which includes all of my local kernel option changes:<\/p>\n
\n
###############################################
\n# \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0local-sysctl.conf \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n# \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 #
\n# Local kernel option settings. \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 #
\n# Install this file in the \/etc\/sysctl.d \u00a0 \u00a0 \u00a0#
\n# directory. \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n# \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 #
\n# Use the command: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n# sysctl -p \/etc\/sysctl.d\/local-sysctl.conf \u00a0 #
\n# to activate. \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n# \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 #
\n###############################################
\n###############################################
\n# Local Network settings \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n# Specifically to disable IPV6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n###############################################
\nnet.ipv6.conf.all.disable_ipv6 = 1
\nnet.ipv6.conf.default.disable_ipv6 = 1

\n###############################################
\n# Virtual Memory \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0#
\n###############################################
\n# Set swappiness
\nvm.swappiness = 1<\/div><\/div><\/pre>\n
I then ran the following command, which activated only the kernel options in the specified file:<\/p>\n
\n
# sysctl -p \/etc\/sysctl.d\/local-sysctl.conf<\/span>
\nnet.ipv6.conf.all.disable_ipv6 = 1<\/span>
\nnet.ipv6.conf.default.disable_ipv6 = 1<\/span>
\nvm.swappiness = 13<\/span><\/div><\/div><\/pre>\n
This is a more targeted approach to setting kernel options than I used in my article about disabling IPv6.<\/p>\n
Reporting the bug<\/h2>\n
At the time of this writing, there is no true fix for the root cause of this problem\u2013whatever the cause. There is a way to temporarily circumvent the issue until a fix is provided. I used the \/etc\/sysctl.d\/local-sysctl.conf<\/code> file that I had created for testing and added a systemd service to run at the end of the startup sequence, wait for a few seconds, and run sysctl<\/code> on that new file. The details of how to do that are in the How I disabled IPv6 on Linux<\/em> article.<\/p>\n
I had already reported this as bug 2103517 using Red Hat’s Bugzilla when trying to disable IPv6. I added this new information to that bug to ensure that my latest findings were available to the kernel developers.<\/p>\n
You can follow the link to view the bug report<\/a>. You do not need an account to view bug reports.<\/p>\n
Final thoughts<\/h2>\n
After experimenting to see how well I could reproduce the symptoms, along with many others, I have determined that the vm.swappiness<\/strong> setting of 60% is far too aggressive for many large-memory Linux systems. Without a lot more data points than those of my own computers, all I can tentatively conclude is that systems with huge amounts of RAM that get used only infrequently are the primary victims of this problem.<\/p>\n
The immediate solution to the problem of local kernel option settings not working is to set them after startup. The automation I implemented is a good example of how to use systemd to replace the old SystemV startup file rc.local<\/code>.<\/p>\n
This bug had not been previously reported. It took a few days of experimenting to verify that the general problem in which locally-set kernel options were not being set or retained at startup time was easily repeatable on multiple physical and virtual systems. At that point, I felt it important to report the bug to ensure it gets fixed. Reporting it is another way I can give back to the Linux community.<\/p>\n<\/div>\n
\n
I recently experienced another interesting problem in the Linux startup sequence that has a circumvention\u2013not a solution. It started quite unexpectedly.<\/p>\n<\/div>\n
\n
\n
  \"Tips<\/div>\n
Image by: <\/span><\/p>\n
opensource.com<\/p>\n<\/div>\n<\/article>\n<\/div>\n
\n
Linux<\/a><\/div>\n
Sysadmin<\/a><\/div>\n<\/p><\/div>\n
What to read next<\/div>\n
\n
How I recovered my Linux system using a Live USB device<\/a><\/div>\n<\/p><\/div>\n

\n        \"Creative<\/a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.<\/div>\n
\n
\n
Register<\/a> or Login<\/a> to post a comment.<\/div>\n<\/p><\/div>\n<\/section>\n
Powered by WPeMatico<\/a><\/small><\/p>\n","protected":false},"excerpt":{"rendered":"
How I troubleshoot swappiness and startup time on Linux David Both Tue, 09\/13\/2022 – 03:00 I recently experienced another interesting problem in the Linux startup sequence that has a circumvention\u2013not a solution. It started quite unexpectedly. I was writing a couple of articles while making some updates to my personal copy of my series of […]<\/p>\n","protected":false},"author":1,"featured_media":67185,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[307],"tags":[],"class_list":["post-67184","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-open-source"],"_links":{"self":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/67184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=67184"}],"version-history":[{"count":0,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/67184\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/media\/67185"}],"wp:attachment":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=67184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=67184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=67184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}