Download Log from AppEngine Including Python Log Statements

Question

I know you can download the raw access logs with appcfg.py, but I'm really interested in all the information around a specific request like python logging statements, exceptions and api statistics (just like the online log viewer). Does anyone know if there is a way to get that information another way then having to build it yourself?

If case anyone is wondering, we want to do some continuos statistical analyzing for problems and displaying them on a large screen on a wall in the office.

Answer 1

Sure - just pass the --severity flag to appcfg.py:

$ appcfg.py help request_logs
Usage: appcfg.py [options] request_logs <directory> <output_file>

Write request logs in Apache common log format.

The 'request_logs' command exports the request logs from your application
to a file.  It will write Apache common log format records ordered
chronologically.  If output file is '-' stdout will be written.

Options:
  -h, --help            Show the help message and exit.
  -q, --quiet           Print errors only.
  -v, --verbose         Print info level logs.
  --noisy               Print all logs.
  -s SERVER, --server=SERVER
                        The server to connect to.
  --insecure            Use HTTP when communicating with the server.
  -e EMAIL, --email=EMAIL
                        The username to use. Will prompt if omitted.
  -H HOST, --host=HOST  Overrides the Host header sent with all RPCs.
  --no_cookies          Do not save authentication cookies to local disk.
  --passin              Read the login password from stdin.
  -A APP_ID, --application=APP_ID
                        Override application from app.yaml file.
  -V VERSION, --version=VERSION
                        Override (major) version from app.yaml file.
  -n NUM_DAYS, --num_days=NUM_DAYS
                        Number of days worth of log data to get. The cut-off
                        point is midnight UTC. Use 0 to get all available
                        logs. Default is 1, unless --append is also given;
                        then the default is 0.
  -a, --append          Append to existing file.
  --severity=SEVERITY   Severity of app-level log messages to get. The range
                        is 0 (DEBUG) through 4 (CRITICAL). If omitted, only
                        request logs are returned.
  --vhost=VHOST         The virtual host of log messages to get. If omitted,
                        all log messages are returned.
  --include_vhost       Include virtual host in log messages.
  --end_date=END_DATE   End date (as YYYY-MM-DD) of period for log data.
                        Defaults to today.

Answer 2

This is what works for us really well:

appcfg.py --append --num_days=0 --include_all request_logs /path/to/your/app/ /var/log/gae/yourapp.log

Anyway, the line above will get all your log records and append them to a log file if you've executed this before, if not, it will create a new log file. It actually looks at your existing log (if it's there) and it will not get any duplicates. You can run this without --append if you want, but use it if you are automating log downloads.

The key here is the --include_allflag which seems to be undocumented. This flag will get all the data that you see if you use GAE's web log viewer. So, you will get fields such as: ms=71 cpu_ms=32 api_cpu_ms=12 cpm_usd=0.000921... etc.

OK, I hope that helps someone.

BTW, we wrote up a blog post on this, check it out here.

Answer 3

I seem to be running into 100M limit with appcfg. I ended up using logservice API to get the logs

Here's the code - https://github.com/manasg/gae-log-fetcher