Downloading detailed logs from google app engine

Question

I want to download the raw access logs from google app engine app written in python.

I have tried following commands:

appcfg.py request_logs --num_days=2 . ~/Documents/workspace/loganalyzer/applog_29march.log

Gives me 47943 records.
But the records does have only header in http clf format. {Here i need to get the detailed log for each record, but i get only headers}

But if i use

appcfg.py --severity=0 --num_days=2 request_logs . ~/Documents/workspace/loganalyzer/applog_27March.log
appcfg.py --severity=1 --num_days=2 --append request_logs . ~/Documents/workspace/loganalyzer/applog_27March.log
appcfg.py --severity=2 --num_days=2 --append request_logs . ~/Documents/workspace/loganalyzer/applog_27March.log
appcfg.py --severity=3 --num_days=2 --append request_logs . ~/Documents/workspace/loganalyzer/applog_27March.log
appcfg.py --severity=4 --num_days=2 --append request_logs . ~/Documents/workspace/loganalyzer/applog_27March.log

I get a total of only 2000 records with detailed errors also.

In my opinion they should match to 47943 records.

Answer 1

App Engine uses circular buffers for the different levels of logs - one for each severity level. When it runs out of space, it overwrites the oldest log entries. As a result, your error logs will go back further than your warning logs, which will go further than info logs, and so forth - and ordinary HTTP request logs will be kept the longest.