Snapd keeps running, causing jbd2/sda2-8 accessing the disk with no read or write, consuming lots of io and system load

Question

I am experiencing high io usage by jbd2/sda2-8 while I am doing nothing.

The problem is that I am running no performance intensive program but the system load continues to be high. Actually the normal load is below 0.05, but from yesterday it has been always been higher than 1.5. After some time digging in the reason, I think that it is the jbd2/sda2-8's io usage that caused the problem.

Later I went to room where the PC locates and found out that the HDD LED light keeps flashing, maybe lots of times in a second. It means that io usage is really a problem.

Here, https://www.webhostingtalk.com/showthread.php?t=1148545, it tells me that jbd2 is not the root cause and I must found out which program is really writing or reading the disk. So I found out that the real cause is snapd.

I tried temporarily stopping the snapd service, and the load went down immediately.

This is Ubuntu Server 20.04 running on an old PC. Here is the system summary:

OS: Ubuntu 20.04 focal
Kernel: x86_64 Linux 5.4.0-33-generic
Uptime: 12h 8m
Packages: 985
Shell: bash 5.0.16
Disk: 11G / 231G (5%)
CPU: Intel Core2 Duo E8600 @ 2x 3.336GHz
GPU: GeForce 9300 GE
RAM: 766MiB / 3935MiB

You can see that it's a dual core cpu so the load 1.5 is really high.

Here is the iotop's feedback

Total DISK READ:         0.00 B/s | Total DISK WRITE:       844.51 K/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:    1643.16 K/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    306 be/3 root        0.00 B/s    0.00 B/s  0.00 % 69.00 % [jbd2/sda2-8]
    972 be/4 root        0.00 B/s  324.81 K/s  0.00 %  0.15 % snapd
    919 be/4 root        0.00 B/s  259.85 K/s  0.00 %  0.12 % snapd
    926 be/4 root        0.00 B/s  259.85 K/s  0.00 %  0.12 % snapd
      1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init maybe-ubiquity
      2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
      3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
      4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_par_gp]
      6 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H-kblockd]
      8 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
      9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
     10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
     11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
     12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [idle_inject/0]
     14 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuhp/0]
     15 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuhp/1]
     16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [idle_inject/1]
     17 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
     18 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
     20 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0H-kblockd]
     21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kdevtmpfs]
     22 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [netns]
     23 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_tasks_kthre]
     24 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kauditd]
     26 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khungtaskd]
  keys:  any: refresh  q: quit  i: ionice  o: active  p: procs  a: accum                                                  
  sort:  r: asc  left: SWAPIN  right: COMMAND  home: TID  end: COMMAND                                                  

The jbd2/sda2-8 is using 69.00% 's io but the speed is zero, just like some previous problem has mentioned. But the difference here is that I am doing nothing, and I don't know which program is causing the problem. Recently I made no large software changes. The only change I made is that I installed and then uninstalled vsftpd.

What I have tried

I have been looking for solutions online, and I have found the follows and tried most of them:

1. Update the kernel. Since I am using the newest OS, I consider it as unnecessary.
2. Change the disk's commit frequency. I am afraid of that and didn't make change to it.
3. Reboot. After reboot the problem is still there.
4. Change the config of mysql. I think mysql is not the reason since my database is small. I turned off mysql and the load immediately dropped to 1.2 and then rise up to 1.6 again.
5. Look for those abnormally large logs and find out what is going on. I find out that auth.log is extremely large and continues to swell up. And I find out that some guy is attacking my server by guessing my root password through ssh. It really shocked me. Huge number of requests from lots of ip addresses! Luckily I denied remote login of root. I requested a new ip address from my ISP and warded off the attack.
6. Run the smart check on my disk. The check passed and my disk is in good condition.
7. Exam the disk usage. You can see from above that there's enough space.

Now that I have temporarily fixed the problem by stopping snapd service

That's all. So what's the problem with snapd and what should I do now?

Answer 1

I uninstalled snapd and the problem was solved.

Answer 2

I have this same problem, my system is ubuntu 20.04 which was just installed. I fixed it by abort snapd job which is running.

1. figure out which job is running by snap changes

this will give you a job number X

2. abort job

snap abort X

3. disable gtk scheme in snap

snap disable ....

It seems only the people behind some kind of POWERFULL firewall or something will have this issue.