AppEngine Response Time Variance

Question

I am considering using AppEngine for deploying a webapp which I am developing. As part of my investigation into the AppEngine platform I have been checking the response time for simple requests. To this end, I have written a simple PING servlet:

@SuppressWarnings("serial")
public class Ping extends HttpServlet
{
  @Override
  public void doGet(@SuppressWarnings("unused") HttpServletRequest xiReq,
                    HttpServletResponse xiResp)
                    throws IOException
  {
    xiResp.setContentType("text/plain");
    xiResp.getWriter().println("PONG");
  }
}

I have then written a java program to make a request every second to this servlet and time how long it takes to complete the request. Fetching the page content uses the following code.

private static String getPageContent(String url) throws IOException {
  String result = null;
  URL reqURL = new URL(url);
  URLConnection connection = reqURL.openConnection();
  connection.setConnectTimeout(30 * 1000);
  connection.setReadTimeout(30 * 3000);
  InputStream webStream =  connection.getInputStream();
  BufferedReader reader = new BufferedReader(new InputStreamReader(webStream));
  result = reader.readLine();
  reader.close();
  return result;
}

Every 3 minutes my monitor script outputs data in the following format:

date,num_reqs,num_failedreqs,avg_reqtime,num_normreqs,avg_normreqtime,num_latereqs,avg_latereqtime

normrequests are all requests which take less than 500ms to complete latereqs are all requests which take longer than 500ms to complete failreqs are any which throw an IO exception during the download or if the content received is not equal to "PONG"

My output for the last ~20 minutes is as follows:

Thu Nov 25 10:04:01 GMT 2010,300,0,186,295,171,5,1093
Thu Nov 25 10:09:28 GMT 2010,300,0,191,292,173,8,842
Thu Nov 25 10:14:52 GMT 2010,300,0,184,295,167,5,1177
Thu Nov 25 10:20:15 GMT 2010,300,0,182,294,168,6,876
Thu Nov 25 10:25:46 GMT 2010,300,0,172,298,167,2,827

This shows that in each 5 minute period there are between 2 and 8 "late" requests taking an average of between 827 and 1177ms to complete.

This compares with the following output from the same period running against the same servlet on a micro instance running on Amazon EC2.

Thu Nov 25 10:03:53 GMT 2010,300,0,177,300,177,0,0
Thu Nov 25 10:09:20 GMT 2010,300,0,179,299,178,1,583
Thu Nov 25 10:14:43 GMT 2010,300,0,176,299,175,1,545
Thu Nov 25 10:20:07 GMT 2010,300,0,176,299,175,1,531
Thu Nov 25 10:25:37 GMT 2010,300,0,181,298,178,2,669

This shows far fewer "late" requests and the response time for these slow requests is much lower.

I am making my requests from a server based in the UK. My Amazon EC2 instance is running in "US East". I don't know where Google is running my AppEngine instance.

Can I do anything to improve the consistency of AppEngine response times or is the variance I am seeing normal for AppEngine?

Answer 1

The 'late' requests you're seeing are due to App Engine spinning up a new Java runtime to handle your request. App Engine increases the number of instances of your app it's running on demand, and spins down idle instances after some period of inactivity.

This behaviour is much more visible for low-traffic apps, because even an individual user can cause a spike that requires new runtimes to be spun up, and instances are more likely to be shut down for inactivity. As the traffic to your app increases, the number of warm-up requests you see will decrease in proportion to the traffic.