what is VIRT column in HTOP? memory overhead?

Question

What means the column VIRT in this HTOP output? It appears in red color and bolded.

I'm running an elasticsearch cluster with 4 nodes and indexing massive data.

• 1700 indexes with 3 shards, some with 10 small docs, others with 500k small docs
• 1 index with 3 shards with 60 million small documents

It shows green health all the time.

This is a DigitalOcean droplet with 4GB RAM and 2 CPU's. I'm setting heapsize to 2gb (-Xms and -Xmx).

Is this an overhead of RAM memory?

  1  [|||||                                9.4%]     Tasks: 26, 122 thr; 2 running   2  [|||                                  4.3%]     Load average: 0.25 0.47 0.65   Mem[||||||||||||||||||||||||||||||2592/3954MB]     Uptime: 2 days, 01:05:57   Swp[                                    0/0MB]    PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command  9629 root       20   0 16.2G 2516M 97360 S 12.0 63.6 27h02:30 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 21564 root       20   0 16.2G 2516M 97360 S  4.0 63.6  1:12.17 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9644 root       20   0 16.2G 2516M 97360 S  1.0 63.6 47:39.34 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 20451 root       20   0 25808  2020  1208 R  0.0  0.0 25:19.48 htop  9654 root       20   0 16.2G 2516M 97360 S  1.0 63.6  5:43.32 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9651 root       20   0 16.2G 2516M 97360 S  0.0 63.6  6:34.53 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9653 root       20   0 16.2G 2516M 97360 S  1.0 63.6  1:46.23 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 21565 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1:12.48 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 21563 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1:11.12 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 21472 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1:15.85 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9652 root       20   0 16.2G 2516M 97360 S  0.0 63.6  5:40.13 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 21562 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1:10.93 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9631 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1h19:18 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9632 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1h19:19 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9633 root       20   0 16.2G 2516M 97360 S  0.0 63.6  7h19:13 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9850 root       20   0 16.2G 2516M 97360 S  0.0 63.6 18:16.00 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9634 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1h29:54 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9722 root       20   0 16.2G 2516M 97360 S  0.0 63.6 50:24.42 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9703 root       20   0 16.2G 2516M 97360 S  0.0 63.6  4h25:50 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9704 root       20   0 16.2G 2516M 97360 S  0.0 63.6  4h26:01 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9662 root       20   0 16.2G 2516M 97360 S  0.0 63.6  1:19.60 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9669 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:08.13 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9665 root       20   0 16.2G 2516M 97360 S  0.0 63.6  6:39.74 /usr/bin/java -Xms256m -Xmx1g -Xss256k -   385 syslog     20   0  243M  1696   872 S  0.0  0.0  0:13.37 rsyslogd -c5   397 syslog     20   0  243M  1696   872 S  0.0  0.0  0:05.58 rsyslogd -c5  9640 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:24.60 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9647 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:10.90 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9635 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:38.98 /usr/bin/java -Xms256m -Xmx1g -Xss256k -   950 root       20   0 15988   664   488 S  0.0  0.0  0:21.99 /usr/sbin/irqbalance  9645 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:19.28 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9700 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:10.25 /usr/bin/java -Xms256m -Xmx1g -Xss256k - 19171 root       20   0 86232  2316  1364 S  0.0  0.1  0:10.56 sshd: root@pts/0  9648 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:09.37 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9639 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:32.61 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9642 root       20   0 16.2G 2516M 97360 S  0.0 63.6  3:04.59 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9643 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:37.35 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9649 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:55.92 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9650 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:15.67 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9706 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:51.00 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9705 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:38.98 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9667 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:08.34 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9646 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:10.10 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9668 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:07.83 /usr/bin/java -Xms256m -Xmx1g -Xss256k -  9666 root       20   0 16.2G 2516M 97360 S  0.0 63.6  0:08.01 /usr/bin/java -Xms256m -Xmx1g -Xss256k -   398 syslog     20   0  243M  1696   872 S  0.0  0.0  0:01.40 rsyslogd -c5   632 root       20   0 50044  1648  1036 S  0.0  0.0  0:02.10 /usr/sbin/sshd -D     1 root       20   0 24300  1644   740 S  0.0  0.0  0:01.80 /sbin/init 

I can buy more machines, but I don't know which is better: another machines VS more RAM.

UPDATE: first and last lines pmap of one process

 root@es2:~# pmap -d 3589  3589:   /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/root/elasticsearch -cp :/root/elasticsearch/lib/elasticsearch-1.1.1.jar:/root/elasticsearch/lib/*:/root/elasticsearch/lib/sigar/* -Xmx2g -Xms2g -Des.node.name=es2 -Des.node.data=true -Des.node.master=false org.elasticsearch.bootstrap.Elasticsearch  Address           Kbytes Mode  Offset           Device    Mapping  0000000000400000       4 r-x-- 0000000000000000 0fd:00000 java  0000000000600000       4 r---- 0000000000000000 0fd:00000 java  0000000000601000       4 rw--- 0000000000001000 0fd:00000 java  000000000188d000     132 rw--- 0000000000000000 000:00000   [ anon ]  0000000775a00000 2118400 rw--- 0000000000000000 000:00000   [ anon ]  00000007f6ec0000   37696 rw--- 0000000000000000 000:00000   [ anon ]  00000007f9390000  111040 rw--- 0000000000000000 000:00000   [ anon ]  00007f6cb0132000    4872 r--s- 0000000000000000 0fd:00000 _8d.fdt  00007f6cb05f4000    5792 r--s- 0000000000000000 0fd:00000 _8d_es090_0.tim  00007f6cb0b9c000    6208 r--s- 0000000000000000 0fd:00000 _8c.fdt  00007f6cb11ac000    6548 r--s- 0000000000000000 0fd:00000 _a0.fdt  00007f6cb1811000    7648 r--s- 0000000000000000 0fd:00000 _a0_es090_0.tim  00007f6cb1f89000    5356 r--s- 0000000000000000 0fd:00000 _78.fdt  00007f6cb24c4000    6236 r--s- 0000000000000000 0fd:00000 _78_es090_0.tim   ....  .......  ....      (huge text)  ....  ......  ..   00007f703dc6e000       4 rw--- 0000000000007000 0fd:00000 librt-2.15.so  00007f703dc6f000      84 r-x-- 0000000000000000 0fd:00000 libgcc_s.so.1  00007f703dc84000    2044 ----- 0000000000015000 0fd:00000 libgcc_s.so.1  00007f703de83000       4 r---- 0000000000014000 0fd:00000 libgcc_s.so.1  00007f703de84000       4 rw--- 0000000000015000 0fd:00000 libgcc_s.so.1  00007f703de85000    1004 r-x-- 0000000000000000 0fd:00000 libm-2.15.so  00007f703df80000    2044 ----- 00000000000fb000 0fd:00000 libm-2.15.so  00007f703e17f000       4 r---- 00000000000fa000 0fd:00000 libm-2.15.so  00007f703e180000       4 rw--- 00000000000fb000 0fd:00000 libm-2.15.so  00007f703e181000     904 r-x-- 0000000000000000 0fd:00000 libstdc++.so.6.0.16  00007f703e263000    2044 ----- 00000000000e2000 0fd:00000 libstdc++.so.6.0.16  00007f703e462000      32 r---- 00000000000e1000 0fd:00000 libstdc++.so.6.0.16  00007f703e46a000       8 rw--- 00000000000e9000 0fd:00000 libstdc++.so.6.0.16  00007f703e46c000      84 rw--- 0000000000000000 000:00000   [ anon ]  00007f703e481000   10988 r-x-- 0000000000000000 0fd:00000 libjvm.so  00007f703ef3c000    2048 ----- 0000000000abb000 0fd:00000 libjvm.so  00007f703f13c000     608 r---- 0000000000abb000 0fd:00000 libjvm.so  00007f703f1d4000     140 rw--- 0000000000b53000 0fd:00000 libjvm.so  00007f703f1f7000     180 rw--- 0000000000000000 000:00000   [ anon ]  00007f703f224000      88 r-x-- 0000000000000000 0fd:00000 libz.so.1.2.3.4  00007f703f23a000    2044 ----- 0000000000016000 0fd:00000 libz.so.1.2.3.4  00007f703f439000       4 r---- 0000000000015000 0fd:00000 libz.so.1.2.3.4  00007f703f43a000       4 rw--- 0000000000016000 0fd:00000 libz.so.1.2.3.4  00007f703f43b000      96 r-x-- 0000000000000000 0fd:00000 libpthread-2.15.so  00007f703f453000    2044 ----- 0000000000018000 0fd:00000 libpthread-2.15.so  00007f703f652000       4 r---- 0000000000017000 0fd:00000 libpthread-2.15.so  00007f703f653000       4 rw--- 0000000000018000 0fd:00000 libpthread-2.15.so  00007f703f654000      16 rw--- 0000000000000000 000:00000   [ anon ]  00007f703f658000       8 r-x-- 0000000000000000 0fd:00000 libdl-2.15.so  00007f703f65a000    2048 ----- 0000000000002000 0fd:00000 libdl-2.15.so  00007f703f85a000       4 r---- 0000000000002000 0fd:00000 libdl-2.15.so  00007f703f85b000       4 rw--- 0000000000003000 0fd:00000 libdl-2.15.so  00007f703f85c000    1748 r-x-- 0000000000000000 0fd:00000 libc-2.15.so  00007f703fa11000    2048 ----- 00000000001b5000 0fd:00000 libc-2.15.so  00007f703fc11000      16 r---- 00000000001b5000 0fd:00000 libc-2.15.so  00007f703fc15000       8 rw--- 00000000001b9000 0fd:00000 libc-2.15.so  00007f703fc17000      20 rw--- 0000000000000000 000:00000   [ anon ]  00007f703fc1c000      52 r-x-- 0000000000000000 0fd:00000 libjli.so  00007f703fc29000    2044 ----- 000000000000d000 0fd:00000 libjli.so  00007f703fe28000       4 r---- 000000000000c000 0fd:00000 libjli.so  00007f703fe29000       4 rw--- 000000000000d000 0fd:00000 libjli.so  00007f703fe2a000     136 r-x-- 0000000000000000 0fd:00000 ld-2.15.so  00007f703fe4c000      32 r--s- 000000000005a000 0fd:00000 lucene-codecs-4.7.2.jar  00007f703fe54000      16 r--s- 0000000000085000 0fd:00000 localedata.jar  00007f703fe58000       8 r--s- 0000000000012000 0fd:00000 zipfs.jar  00007f703fe5a000      12 r--s- 0000000000032000 0fd:00000 sunjce_provider.jar  00007f703fe5d000      16 r--s- 000000000003b000 0fd:00000 sunpkcs11.jar  00007f703fe61000     556 rw--- 0000000000000000 000:00000   [ anon ]  00007f703feec000      72 rw--- 0000000000000000 000:00000   [ anon ]  00007f703fefe000     220 rw--- 0000000000000000 000:00000   [ anon ]  00007f703ff35000      40 rw--- 0000000000000000 000:00000   [ anon ]  00007f703ff3f000     148 rw--- 0000000000000000 000:00000   [ anon ]  00007f703ff64000     580 rw--- 0000000000000000 000:00000   [ anon ]  00007f703fff5000      32 rw-s- 0000000000000000 0fd:00000 3279  00007f703fffd000      12 ----- 0000000000000000 000:00000   [ anon ]  00007f7040000000     268 rw--- 0000000000000000 000:00000   [ anon ]  00007f7040043000       4 r--s- 0000000000008000 0fd:00000 lucene-memory-4.7.2.jar  00007f7040044000       8 r--s- 000000000000f000 0fd:00000 pulse-java.jar  00007f7040046000       4 r--s- 0000000000007000 0fd:00000 java-atk-wrapper.jar  00007f7040047000       4 r--s- 0000000000002000 0fd:00000 dnsns.jar  00007f7040048000       4 rw--- 0000000000000000 000:00000   [ anon ]  00007f7040049000       4 r---- 0000000000000000 000:00000   [ anon ]  00007f704004a000       8 rw--- 0000000000000000 000:00000   [ anon ]  00007f704004c000       4 r---- 0000000000022000 0fd:00000 ld-2.15.so  00007f704004d000       8 rw--- 0000000000023000 0fd:00000 ld-2.15.so  00007fff9dcbe000     132 rw--- 0000000000000000 000:00000   [ stack ]  00007fff9ddfe000       8 r-x-- 0000000000000000 000:00000   [ anon ]  ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]  ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]  mapped: 17052740K    writeable/private: 2523144K    shared: 13562836K  root@es2:~# 

Answer 1

I would stop worrying about the VIRT thing.

Virtual Memory Usage from Java under Linux, too much memory used

I suspect the top page is allocated for vsyscall purposes (see What are vdso and vsyscall?). Meaning that whilst your process has a 16Gb range of virtual memory mapped, it certainly isn't using anything like that in terms of physical memory. Go by the RES entry.

Regarding whether you need more machines or more memory... Is it slow? You would need to do more forensics on where your bottlenecks are, but the htop capture there doesn't appear to be under much pressure.