Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

PHP factor 30 performance difference from Linux to Windows

Our team is working is developing WordPress plugins and provides hosted instances on a couple of independent servers. Our WordPress installation is managed by Git, all servers have the same source & WordPress setup deployed, only domains & actual data in the database varies. For each installation, MySql is running on the same host. WordPress is running exclusively on each server.

However after having deployed this setup on a Windows Server 2008 RC2, we noticed a drastic performance difference compared to our other servers: page generation time goes up from avg. 400ms to 4000-5000ms for pages generated with PHP. For static resources delivered by Apache only, speed is about the same as on linux.

So we took some steps to narrow down the problem:

  1. Make sure there is no antivir-software running or other windows domain stuff interfering
  2. Collect profiling data to identify the timekillers during script execution
  3. Test different server & hardware setups
  4. Double-check both Apache and PHP configuration for obvious configuration errors

After some profiling we quickly noticed that the evaluation of regular expressions is horribly slow on our windows machines. Evaluating 10.000 Regular expressions (preg_match) takes about 90ms on Linux and 3000ms on Windows.

Profiling, system tests and configuration details are provided bellow. We don't want to optimize this script (which we do know how to do). We want to get the script to run approximately the same speed on windows as on Linux (given the same setup regarding opcache/...). No need to optimize the memory footprint of the script too.

Update: After some time, the systems seems to run out of memory, triggering out of memory exceptions and random allocations. See bellow for more details. Restarting Apache/PHP fixed the problem for now.

Trace to _get_browser is:

File (called from) require wp-blog-header.php (index.php:17) wp (wp-blog-header.php:14) WP->main (functions.php:808) php::do_action_ref_array (class-wp.php:616) php::call_user_func_array (wp-includes/plugin:507) wp_slimstat::slimtrack  (php::internal (507)) wp_slimstat::_get_browser (wp-slimstat.php:385) 

Update 2: Some some reason I can't remember we went back to activating PHP as an Apache Module on our servers (the same which deliver bad performance). But today they run blazingly fast (~1sec/request). Adding Opcache brings this down to ~400ms/req. Apache/PHP/Windows remained the same.

1) Profiling Results

Profiling was done with XDebug on all machines. Usually we only collected a few runs - those were enough to reveal the location where most of the time (50%+) was spent: the method [get_browser][1] of the WordPress plugin wp-slimstats:

protected static function _get_browser(){     // Load cache     @include_once(plugin_dir_path( __FILE__ ).'databases/browscap.php');     // browscap.php contains $slimstat_patterns and $slimstat_browsers      $browser = array('browser' => 'Default Browser', 'version' => '1', 'platform' => 'unknown', 'css_version' => 1, 'type' => 1);     if (empty($slimstat_patterns) || !is_array($slimstat_patterns)) return $browser;      $user_agent = isset($_SERVER['HTTP_USER_AGENT'])?$_SERVER['HTTP_USER_AGENT']:'';     $search = array();     foreach ($slimstat_patterns as $key => $pattern){         if (preg_match($pattern . 'i', $user_agent)){             $search = $value = $search + $slimstat_browsers[$key];             while (array_key_exists(3, $value) && $value[3]) {                 $value = $slimstat_browsers[$value[3]];                 $search += $value;             }             break;         }     }      // Lots of other lines to relevant to the profiling results   } 

This function similar to PHP's get_browser detects the browser's capabilities and OS. Most of the script execution time is spent in this foreach loop, evaluating all those preg_match (~approx 8000 - 10000 per page request). This takes about 90ms on Linux and 3000ms on Windows. Results were the same on all setups tested (picture shows data of two executions):

wp_slimstat::_get_browser profiling results on IIS8

Sure, loading two huge arrays takes some time. Evaluating regular expressions too. But we'd expect them to take approximately the same time on Linux and Windows. This is the profiling result on a linux vm (one page request only). The difference is pretty obvious:

enter image description here

Another time killer was actually the Object-Cache WordPress uses:

function get( $key, $group = 'default', $force = false, &$found = null ) {     if ( empty( $group ) )         $group = 'default';      if ( $this->multisite && ! isset( $this->global_groups[ $group ] ) )         $key = $this->blog_prefix . $key;      if ( $this->_exists( $key, $group ) ) {         $found = true;         $this->cache_hits += 1;         if ( is_object($this->cache[$group][$key]) )             return clone $this->cache[$group][$key];         else             return $this->cache[$group][$key];     }      $found = false;     $this->cache_misses += 1;     return false; } 

Time is spent within this function itself (3 script executions):

enter image description here

On linux:

enter image description here

The last real big time killer were translations. Each translation, loaded from memory, takes anything from 0.2ms to 4ms in WordPress: enter image description here

On linux:

enter image description here

2) Tested systems

In order to make sure virtualization or Apache do affect this, we tested this on several setups. Antivir was disabled on all setups:

  • Linux Debian, Apache 2 & PHP on up to date stable releases. This is the same for developers running in their virtual machines as for staging/live servers. Acting as a reference system of desired performance. Either run in our office or at some hosting provides (shared space). Windows Systems had between 4GB and 8GB of RAM, at all time memory usage was bellow 50%. Virtualizations never run Windows & Apache at the same time.
  • Life-Servers, running at T-Systems (managed virtualized servers), on VMWare Player
    • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module
    • Win 2008 R2. Apache 2.2.25 + PHP 5.5.1 NTS,VC11 as fastcgi module
    • Win 2008 R2. Apache 2.2.25 + PHP 5.5.1 NTS,VC11 as apache module
    • Win 2008 R2, Apache 2.2.25 + PHP 5.5.11 TS,VC11 as apache module (that's the fast one I mentioned in the update 2)
  • On a local machine, Host: OpenSuse, Virtualization: VMWare player, same as @T-Systems. To avoid their infrastructure influencing us:
    • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module
    • Win 2008 R2. IIS7 + PHP 5.4.26 NTS,VC9 as fastcgi module (with and without wincache)
    • Win 2012. IIS * + PHP 5.5.10 NTS,VC11 as fastcgi module (with and without wincache)
  • On a local machine without virtualization
    • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module

Profiling results as mentioned above were the same on the different systems (~10% derivation). Windows was always a significant factor slower then Linux.

Using a fresh install of WordPress & Slimstats resulted in approx. the same results. Rewriting the code is not an option here.

Update: Meanwhile we found two other Windows Systems (both Windows 2008 R2, VM & Phys) where this complete stack runs quite fast. Same configuration though.

Update 2: Running PHP as apache module on the Life-Servers was slightly faster then the fastcgi method: down to ~2sec, 50% less.

Running out of Memory

After some time, our Live-Server stops working at all, triggering these out of memory exceptions:

PHP Fatal error:  Out of memory (allocated 4456448) (tried to allocate 136 bytes) PHP Fatal error:  Out of memory (allocated 8650752) (tried to allocate 45 bytes)  PHP Fatal error:  Out of memory (allocated 6815744) (tried to allocate 24 bytes)  

This happens at random script locations. Obviously the Zend Memory Manager is not able to allocate more memory, although the scripts would be allowed to do so. At the time if incident, the server had about 50% of free RAM (2GB+). So the server does not actually run out of ram. Restarting Apache/PHP fixed this problem for now.

Not sure if this problem is related to the performance issues here. Yet as both issues seem to be memory related, its included here. Especially we'll try to reproduce the settings of the Windows-Tests that provided decent performance.

3) Apache & PHP Configuration

... probably do not have any common pitfalls. Output-Buffering is enabled (to default), multibye override disabled, ... If any option(s) are of interest we'll happily provide them.

Output of httpd.exe -V

Server version: Apache/2.4.7 (Win32) Apache Lounge VC10 Server built:   Nov 26 2013 15:46:56 Server's Module Magic Number: 20120211:27 Server loaded:  APR 1.5.0, APR-UTIL 1.5.3 Compiled using: APR 1.5.0, APR-UTIL 1.5.3 Architecture:   32-bit Server MPM:     WinNT   threaded:     yes (fixed thread count)     forked:     no Server compiled with....  -D APR_HAS_SENDFILE  -D APR_HAS_MMAP  -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)  -D APR_HAS_OTHER_CHILD  -D AP_HAVE_RELIABLE_PIPED_LOGS  -D DYNAMIC_MODULE_LIMIT=256  -D HTTPD_ROOT="/apache"  -D SUEXEC_BIN="/apache/bin/suexec"  -D DEFAULT_PIDLOG="logs/httpd.pid"  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"  -D DEFAULT_ERRORLOG="logs/error.log"  -D AP_TYPES_CONFIG_FILE="conf/mime.types"  -D SERVER_CONFIG_FILE="conf/httpd.conf" 

mpm_winnt_module configuration:

<IfModule mpm_winnt_module>     ThreadsPerChild 150     ThreadStackSize 8388608      MaxConnectionsPerChild 0 </IfModule> 

Excerpt of php.ini:

realpath_cache_size = 12M pcre.recursion_limit = 100000 

4) Current suspected reason

Old assumption:

All three examples heavily rely on big arrays and string operations. That some kind seems to be the common factory. As the implementation works ok'ish on Linux, we suspect this to be a memory problem on Windows. Given there is no database interaction at the pin-pointed locations, we don't suspect the database or Server <-> PHP integration to be the problem. Somehow PHP's memory interaction just seems to be slow. Maybe there is someone interfering with the memory on Windows making access dramatically slower?

Old assumption 2:

As the same stack runs fine on other Windows machines we assume the problem to be somewhere in the Windows configuration.

New assumption 3:

Actually I am out of assumptions. Why would run PHP that much slower as fastcgi then as apache module>

Any ideas on how to verify this or finding the real problem in here? Any help or direction for fixing this issue is highly welcome.

like image 674
Fge Avatar asked Apr 03 '14 17:04

Fge


1 Answers

Windows has lots of services/policies that restrict, prevent, protect, control and etc usage of the computer in every situation.

A good Microsoft certified specialist will be able to solve your question within minutes, because they will have the experience to tell exactly which settings/services/policies to check and disable/enable/change settings, so that the PHP scripts are executed faster.

Out of my memory, I can only suggest you to check everything that deals with RAM, Hard Drive access, Environmental variables, Limits and Security (like Firewall). Everything that can affect the execution of php script, starting with some Remote Procedue Call policies and ending with the operating stack memory.

The logic is that is php.exe calls some external .dll file to execute some operation, there might be checks on the way done by OS, that will slow both sending request via such .dll, and receiving the response from it. If the .dll uses hard drive to access something - hard drive access policies enter into the scene. Also, how everything is situated in the memory - in RAM or hard-drive cache of RAM. Application policies. Threads policies. Limits on max percentage available for use for applications.

I am not saying that Windows-based hosts are bad, just that they are much more difficult to setup properly for a general admin. If you have Microsoft specialist on hands, he can tune your server to be as fast as Linux-based server.

like image 62
Anatoliy Kim Avatar answered Oct 04 '22 04:10

Anatoliy Kim