Why is the system CPU time (% sy) high?

Question

I am running a script that loads big files. I ran the same script in a single core OpenSuSe server and quad core PC. As expected in my PC it is much more faster than in the server. But, the script slows down the server and makes it impossible to do anything else.

My script is

for 100 iterations
Load saved data (about 10 mb)

time myscript (in PC)

real    0m52.564s
user    0m51.768s
sys    0m0.524s

time myscript (in server)

real    32m32.810s
user    4m37.677s
sys    12m51.524s

I wonder why "sys" is so high when i run the code in server. I used top command to check the memory and cpu usage. It seems there is still free memory, so swapping is not the reason. % sy is so high, its probably the reason for the speed of server but I dont know what is causing % sy so high. The process that is using highest percent of CPU (99%) is "myscript". %wa is zero in the screenshot but sometimes it gets very high (50 %).

When the script is running, load average is greater than 1 but have never seen to be as high as 2.

I also checked my disc:

strt:~ # hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   16480 MB in  2.00 seconds = 8247.94 MB/sec
 Timing buffered disk reads:   20 MB in  3.44 seconds =   5.81 MB/sec

john@strt:~> df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       245G  102G  131G  44% /
udev            4.0G  152K  4.0G   1% /dev
tmpfs           4.0G   76K  4.0G   1% /dev/shm

I have checked these things but I am still not sure what is the real problem in my server and how to fix it. Can anyone identify a probable reason for the slowness? What could be the solution? Or is there anything else I should check?

Thanks!

Answer 1

You're getting a high sys activity because the load of the data you're doing takes system calls that happen in kernel. To resolve your slowness problems without upgrading the server might be possible. You can modify scheduling priority. See the man pages for nice and renice. See here and especially:

Niceness values range from -20 (the highest priority, lowest niceness) and 19 (the lowest priority, highest niceness).

$ ps -lp 941
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 941 1 0 70 -10 - 1713 poll_s ? 00:00:00 sshd

$ nice -n 19 ./test.sh 
My niceness value is 19

$ renice -n 10 -p 941
941 (process ID) old priority -10, new priority 10