Why is there such a big difference in the transfer rates of the Java FTP Clients

Question

maybe someone can answer me that question. During my recent works I noticed that my application (which downloads updates via FTP) was really slow when used with Linux. I develop that stuff on a Mac so I didn't notice that issue earlier, because the download speed didn't feel very low under Mac OS. But when moving to Linux the application behaved really different.

The FTP server (Pure FTP running on Ubuntu Server) is connected to the same LAN as the clients, so internet speed issues do not come into account. Because of the low performance I changed the Apache FTPClient to edtFTPj/Free. The differences are still remarkable but acceptable. As a test case I downloaded always the same file which had a size of about 30 MB. Then I checked the ftp server log to find out about the transfer rate.

See for Yourself. The VMWare mentioned runs on the Mac. The Java is a Oracle Java 1.7 unless noted otherwise.

Apache Commons Net 2.3

The code looks like this

  FTPClient ftp = new FTPClient();
  ftp.connect("srv0006");
  ftp.login("anonymous", "asd");
  ftp.setFileType(FTP.BINARY_FILE_TYPE);
  File target = new File("/tmp/pub.tar");
  FileOutputStream fos = new FileOutputStream(target);
  ftp.retrieveFile("/pub.tar", fos);
  fos.close();

Here are the results from the ftp log

Mac OS started from IntelliJ Idea

downloaded  (30452736 bytes, 21200.67KB/sec)

Mac OS started from shell

downloaded  (30452736 bytes, 21471.75KB/sec)

Windows 7 (in VMWare)

downloaded  (30452736 bytes, 65243.15KB/sec)

OpenSuse running Oracle Java (in VMWare)

downloaded  (30452736 bytes, 5274.56KB/sec)

OpenSuse running OpenJDK (in VMWare)

downloaded  (30452736 bytes, 7663.68KB/sec)

Ubuntu 12.04.1 LTS

running on another PC connected with Gigabit Ethernet to the same LAN. Other Ubuntu Machines behaved exactly the same way. I quit the transfer after 20 Minutes. See the transfer rate.

downloaded  (7077888 bytes, 6.10KB/sec)

edtFTP4j 2.4.0

After this, I moved to edtFTP4j. The results were much better.

  FileTransferClient ftp = new FileTransferClient();
  ftp.setRemoteHost("srv0006");
  ftp.setUserName("anonymous");
  ftp.setPassword("asd");
  ftp.connect();
  ftp.downloadFile("/tmp/pub.tar", "/pub.tar");
  ftp.disconnect();

The results changed remarkably:

Mac OS started from IntelliJ Idea

downloaded  (30452736 bytes, 109431.60KB/sec)

Mac OS started from shell

downloaded  (30452736 bytes, 110333.66KB/sec)

Windows 7 (in VMWare)

downloaded  (30452736 bytes, 91318.64KB/sec)

OpenSuse running Oracle Java (in VMWare)

downloaded  (30452736 bytes, 89312.46KB/sec)

OpenSuse running OpenJDK (in VMWare)

downloaded  (30452736 bytes, 89041.05KB/sec)

Ubuntu 12.10 (in VMWare)

downloaded  (30452736 bytes, 81154.99KB/sec)

Ubuntu 12.04.1 LTS running on i5 Notebook, Wifi (50 MBit/s)

downloaded  (30452736 bytes, 2883.84KB/sec)

Ubuntu 12.04.1 LTS running on i5 Notebook, Gigabit Ethernet

downloaded  (30452736 bytes, 93822.44KB/sec)

Ubuntu 12.04.1 LTS

running on the PC mentioned earlier (the one with the 6.10 KB/sec tx rate)

downloaded  (30452736 bytes, 11633.38KB/sec)

I don't understand this. Who has a clue whats going on here ?

Bye, Torsten...

Answer 1

There's too much going on to pinpoint it from some code.

It could be anything from your default network packet size (MTU), your hardware infrastructure, your JVM, your OS configuration, etc..

You need to play with lots of little things at the higher level. Things like checking your MTU size on the hardware/OS level and correspond that to the default settings for the API's socket creation. Does your infrastructure buffer or have window scaling or do automated virus checking.

I believe Net Commons buffer size defaults to 1024, you can play with that.

Beyond that, you need to drop down into the sniffer and see what's going on. Could be a switch isn't configured correctly and works better with one API vs. the other.

Wish I could give you a better answer, but when it comes to networking performance, it's really a field of study in and of itself...