App Engine downtime

Question

Does Google's App Engine have excessive downtime, specifically with regards to datastore writes?

Additionally, downtime seems to be scheduled during high traffic times, e.g., in the middle of the afternoon vs. 3:00AM in the morning. Is this normal? Will it improve as the technology matures?

Answer 1

Short Answers

1. Afternoon vs early morning downtime. The datastore has been unavailable for writes about 20-30% more often in the afternoon than in the wee hours of the morning (Pacific time; includes put, update, and delete availability).

   Note: I'm sure Google would like downtime to occur during off-peak hours. Thus I expect they'll continue to try to minimize downtime, or schedule it for off-peak hours whenever possible.

2. Downtime trending. The number of 15-minute periods during which the datastore has been unavailable has been decreasing. In the past 366 days, there were an average of 3.8 15-minute periods in which the datastore was unavailable per day. In the past 200 days, this has decreased by 60% to 2.3 per day. Write downtime over the past few months has actually been quite good - since March 1st, there have been less than 0.25 15-minute chunks of write downtime per day. Here's a graph of datastore write downtime: Downtime trending http://imagebin.ca/img/4wkHVQPc.png

---

Source of Answers

To answer your question, I wrote this script which extracts downtime data from GAE's Datastore Status page.

---

Graphs

Datastore write downtime from 2009-Jul-20 to 2010-Jul-20 (4 hour bins):

alt text http://imagebin.ca/img/p9ScWTm.png

Datastore write downtime from 2009-Jul-20 to 2010-Jul-20 (1 hour bins):

alt text http://imagebin.ca/img/9FbLut2G.png

Datastore downtime from 2009-Jul-20 to 2010-Jul-20 (4 hour bins):

alt text http://imagebin.ca/img/t3XKLk.png

Datastore downtime from 2010-Jan-01 to 2010-Jul-20 (4 hour bins):

alt text http://imagebin.ca/img/k36T9h.png

---

Raw data

(you can tweak the variables at the top of the script if you'd like to collect your own data with slightly different parameters):

# RAW Data: Each element counts the number of days in which the datastore
# was unavailable for at least some portion of a given 15-minute window. The
# first element corresponds to the time chunk from 00:00 to 00:15, and so on.
RESULTS_SINCE_2010JAN01_BIN15 = [0, 0, 0, 0, 3, 11, 3, 3, 3, 3, 12, 3, 3, 3, 4, 14, 4, 4, 4, 4, 12, 2, 2, 2, 2, 14, 4, 4, 4, 4, 11, 2, 2, 2, 2, 11, 5, 5, 5, 5, 13, 4, 4, 4, 4, 14, 7, 5, 5, 5, 14, 4, 3, 3, 3, 13, 2, 2, 2, 2, 12, 5, 4, 4, 4, 14, 5, 3, 3, 3, 12, 7, 2, 2, 2, 5, 5, 0, 0, 0, 2, 9, 3, 2, 2, 2, 10, 1, 1, 1, 2, 9, 3, 3, 3, 15]
RESULTS_SINCE_2009JUL20_BIN15 = [0, 0, 0, 0, 11, 21, 5, 5, 5, 5, 29, 6, 6, 6, 7, 38, 11, 11, 11, 11, 37, 7, 7, 7, 7, 44, 12, 12, 12, 12, 37, 10, 10, 10, 10, 34, 7, 7, 7, 7, 46, 11, 11, 11, 11, 39, 15, 13, 13, 13, 44, 13, 12, 12, 12, 44, 5, 5, 5, 5, 34, 11, 10, 10, 10, 40, 13, 11, 11, 11, 31, 21, 12, 12, 11, 19, 21, 4, 4, 4, 13, 28, 10, 9, 9, 16, 36, 10, 10, 10, 12, 32, 7, 7, 6, 35]
RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN15 = [0, 0, 0, 0, 4, 12, 4, 4, 4, 4, 22, 6, 6, 6, 7, 27, 7, 7, 7, 7, 21, 6, 6, 6, 6, 32, 9, 9, 9, 9, 26, 8, 8, 8, 8, 27, 7, 7, 7, 7, 30, 7, 7, 7, 7, 27, 10, 8, 8, 8, 28, 10, 9, 9, 9, 28, 4, 4, 4, 4, 21, 4, 4, 4, 4, 25, 6, 4, 4, 4, 18, 14, 9, 10, 9, 16, 17, 2, 2, 2, 8, 18, 7, 6, 6, 9, 19, 5, 5, 5, 6, 18, 5, 5, 4, 21]

# RESULTS DISTILLED FROM COLLECTED_RESULTS
RESULTS_SINCE_2010JAN01_BIN60 = [RESULTS_SINCE_2010JAN01_BIN15[i*4]+RESULTS_SINCE_2010JAN01_BIN15[i*4+1]+RESULTS_SINCE_2010JAN01_BIN15[i*4+2]+RESULTS_SINCE_2010JAN01_BIN15[i*4+3] for i in xrange(24)]
RESULTS_SINCE_2010JAN01_BIN240 = [RESULTS_SINCE_2010JAN01_BIN60[i*4]+RESULTS_SINCE_2010JAN01_BIN60[i*4+1]+RESULTS_SINCE_2010JAN01_BIN60[i*4+2]+RESULTS_SINCE_2010JAN01_BIN60[i*4+3] for i in xrange(6)]
RESULTS_SINCE_2010JAN01_BIN480 = [RESULTS_SINCE_2010JAN01_BIN60[i*2]+RESULTS_SINCE_2010JAN01_BIN60[i*2+1] for i in xrange(3)]
RESULTS_SINCE_2009JUL20_BIN60 = [RESULTS_SINCE_2009JUL20_BIN15[i*4]+RESULTS_SINCE_2009JUL20_BIN15[i*4+1]+RESULTS_SINCE_2009JUL20_BIN15[i*4+2]+RESULTS_SINCE_2009JUL20_BIN15[i*4+3] for i in xrange(24)]
RESULTS_SINCE_2009JUL20_BIN240 = [RESULTS_SINCE_2009JUL20_BIN60[i*4]+RESULTS_SINCE_2009JUL20_BIN60[i*4+1]+RESULTS_SINCE_2009JUL20_BIN60[i*4+2]+RESULTS_SINCE_2009JUL20_BIN60[i*4+3] for i in xrange(6)]
RESULTS_SINCE_2009JUL20_BIN480 = [RESULTS_SINCE_2009JUL20_BIN240[i*2]+RESULTS_SINCE_2009JUL20_BIN240[i*2+1] for i in xrange(3)]
RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN60 = [RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN15[i*4]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN15[i*4+1]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN15[i*4+2]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN15[i*4+3] for i in xrange(24)]
RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN240 = [RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN60[i*4]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN60[i*4+1]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN60[i*4+2]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN60[i*4+3] for i in xrange(6)]
RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN480 = [RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN240[i*2]+RESULTS_WRITE_DOWNTIME_SINCE_2009JUL20_BIN240[i*2+1] for i in xrange(3)]