Menu
I'm using python and requests module==2.18.4
While crawling some data with requests, I used the logging module for debugging.
I want the log to look something like this:
[DEBUG] 2018-01-25 03:15:36,940 http://localhost:8888 "GET /aaa" 200 2290
[DEBUG] 2018-01-25 03:15:36,940 http://localhost:8888 "GET /aaa" 200 2290
[DEBUG] 2018-01-25 03:15:36,940 http://localhost:8888 "GET /aaa" 200 2290
But I get this:
[DEBUG] 2018-01-25 03:15:36,940 http://localhost:8888 "GET /aaa" 200 2290
[DEBUG] 2018-01-25 03:15:36,974 EUC-JP Japanese prober hit error at byte 1765
[DEBUG] 2018-01-25 03:15:36,990 EUC-KR Korean prober hit error at byte 1765
[DEBUG] 2018-01-25 03:15:36,994 CP949 Korean prober hit error at byte 1765
[DEBUG] 2018-01-25 03:15:37,009 EUC-TW Taiwan prober hit error at byte 1765
[DEBUG] 2018-01-25 03:15:37,036 utf-8 not active
[DEBUG] 2018-01-25 03:15:37,036 SHIFT_JIS Japanese confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,036 EUC-JP not active
[DEBUG] 2018-01-25 03:15:37,036 GB2312 Chinese confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,036 EUC-KR not active
[DEBUG] 2018-01-25 03:15:37,036 CP949 not active
[DEBUG] 2018-01-25 03:15:37,036 Big5 Chinese confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,036 EUC-TW not active
[DEBUG] 2018-01-25 03:15:37,036 windows-1251 Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 KOI8-R Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 ISO-8859-5 Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 MacCyrillic Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 IBM866 Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 IBM855 Russian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 ISO-8859-7 Greek confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 windows-1253 Greek confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 ISO-8859-5 Bulgairan confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 windows-1251 Bulgarian confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 TIS-620 Thai confidence = 0.01
[DEBUG] 2018-01-25 03:15:37,038 ISO-8859-9 Turkish confidence = 0.47949350706
[DEBUG] 2018-01-25 03:15:37,038 windows-1255 Hebrew confidence = 0.0
[DEBUG] 2018-01-25 03:15:37,038 windows-1255 Hebrew confidence = 0.0
[DEBUG] 2018-01-25 03:15:37,038 windows-1255 Hebrew confidence = 0.0
...
I don't want that encoding in logs. How can I remove them?
763
Python comes with a logging module in the standard library that provides a flexible framework for emitting log messages from Python programs. This module is widely used by libraries and is the first go-to point for most developers when it comes to logging.
logger = logging.getLogger(__name__) This means that logger names track the package/module hierarchy, and it's intuitively obvious where events are logged just from the logger name. Sounds like good advice.
Python logging basicConfig The basicConfig configures the root logger. It does basic configuration for the logging system by creating a stream handler with a default formatter. The debug , info , warning , error and critical call basicConfig automatically if no handlers are defined for the root logger.
Try setting the logging level for the module chardet.charsetprober to something higher than DEBUG (eg. INFO).
logger = logging.getLogger('chardet.charsetprober')
logger.setLevel(logging.INFO)
I had the same issue and found these the extra logs coming from chardet.charsetprober module.
To suppress these logs, put this after imports.
logging.getLogger('chardet.charsetprober').setLevel(logging.INFO)
This won't print any DEBUG level message from chardet.charsetprober module and you will get the desired log message only.
Hope it helps!
42
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
