ApplicationError: 7 when processing is done for mapreduce worker writing to Google Cloud Storage

Question

Just finishing up a large Appengine mapreduce task, and many of my shards are stuck moments from the finishline. Here's the setup:

    filenames = yield mapreduce_pipeline.MapperPipeline(
            'example mapper name',
            'main.MyMapper',
            input_reader_spec='mapreduce.input_readers.DatastoreInputReader',
            output_writer_spec='mapreduce.output_writers.FileOutputWriter',
            params={
                'input_reader':{
                    'entity_kind':'models.MyModel'
                },
                'output_writer':{
                    'filesystem':'gs',
                    'mime_type':'text/csv',
                    'gs_bucket_name':'myBucket',
                    'output_sharding':'input'
                }
            },
            shards=DUMP_SHARDS
            )

I'm running 3 of these in parallel, each with 16 shards. One mapper completed with no issue, the other two mappers have had success on 14 and 9 of their shards.

The remaining shards are all completely stonewalled, returning UnknownError: ApplicationError: 7. (Full stack trace at the end of this post.)

Note that the mapper is trying to write to Google Cloud Storage. The error occurs in the bits which are performing this write.

After hunting around for a while, I found, in google.appengine.runtime.apiproxy (which appears to be the proxy in question), that error 7 is OTHER_ERROR.

I've been retrying these final tasks (from the task queue) for about 3 hours now, and not one has succeeded since these errors began; whatever is happening, it's totally stuck. I've also tried stopping all the instances this was running on, in case it was some weird local state thing, but no change there...

Here is the full stack trace:

I 2012-12-13 15:40:23.909
Processing done for shard 14 of job '1582444192075C233F6AA'
E 2012-12-13 15:40:23.969
ApplicationError: 7 
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/base_handler.py", line 65, in post
    self.handle()
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/handlers.py", line 231, in handle
    tstate.output_writer.finalize(ctx, shard_state.shard_number)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/output_writers.py", line 631, in finalize
    files.finalize(self._filename)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 568, in finalize
    f.close(finalize=True)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 291, in close
    self._make_rpc_call_with_retry('Close', request, response)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 427, in _make_rpc_call_with_retry
    _make_call(method, request, response)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 252, in _make_call
    _raise_app_error(e)
  File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 186, in _raise_app_error
    raise UnknownError(e)
UnknownError: ApplicationError: 7 

Answer 1

I've just had a similar issue. I think it's specifically a writing to Google Cloud Storage problem.

I've gained some insight here: Google App Engine Issue: 8775

Summary (TLDR):

• Could be a one-time network issue.
• Could be a billing issue.
• Result: If it doesn't go away and fixing billing doesn't work, Contact Google Support.