Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Turning off Snowflake DB logging while still keeping log level as DEBUG

Is it possible to disable the Snowflake SQL logging which logs the start and end of a DB connection and all the queries being executed while keeping logging.basicConfig(level=logging.INFO) for debugging my development.

Maybe this is not specific to Snowflake but a general DB connection from Python?

I'm using snowflake-connector-python version 1.8.1

Sample simplified log that I currently have.

2019-06-10 16:27:10,015 INFO: /*Need this line*/

2019-06-10 16:27:10,015 INFO: Snowflake Connector for Python Version: 1.8.1, Python Version: 3.7.3, Platform: Windows-7-6.1.7601-SP1
2019-06-10 16:27:10,015 INFO: This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
2019-06-10 16:27:10,020 INFO: Starting new HTTPS connection (1): xyz.snowflakecomputing.com
2019-06-10 16:27:11,227 INFO: query: [USE WAREHOUSE test_wh]
2019-06-10 16:27:11,481 INFO: query execution done
2019-06-10 16:27:11,481 INFO: query: [SELECT COLUMN_NAME FROM DB.INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG =...]
2019-06-10 16:27:12,830 INFO: query execution done
2019-06-10 16:27:12,830 INFO: fetching data done
2019-06-10 16:27:12,830 INFO: closed
2019-06-10 16:27:13,185 INFO: /*Need this line*/
2019-06-10 16:27:13,581 INFO: /*Need this line*/
2019-06-10 16:27:14,604 INFO: /*Need this line*/
like image 673
Vinay Avatar asked Jun 10 '19 11:06

Vinay


2 Answers

In your Python module that imports snowflake.connector, before you call any methods on the connector, add

logging.getLogger('snowflake.connector').setLevel(logging.WARNING)

to have it log only warnings or higher levels.

like image 71
John Velonis Avatar answered Oct 09 '22 22:10

John Velonis


I find that @john Velonis' solution doesn't quite get rid of all snowflake messages. I was still getting messages like this:

2020-03-26 18:03:04    DEBUG:  network._request_exec:805 - SUCCESS
2020-03-26 18:03:04    DEBUG:  network._use_requests_session:944 - Active requests sessions: 0, idle: 1
2020-03-26 18:03:04    DEBUG:  network._post_request:533 - ret[code] = None, after post request
2020-03-26 18:03:04    DEBUG:   cursor.execute:543 - sfqid: 0193261b-0310-8c37-0003-c9033ae3696a
2020-03-26 18:03:04     INFO:   cursor.execute:545 - query execution done
2020-03-26 18:03:04    DEBUG:   cursor.execute:547 - SUCCESS
2020-03-26 18:03:04    DEBUG:   cursor.execute:551 - PUT OR GET: None
2020-03-26 18:03:04    DEBUG:   cursor._init_result_and_meta:611 - Query result format: arrow
2020-03-26 18:03:04    DEBUG:   cursor._init_result_and_meta:631 - Batches read: 8
2020-03-26 18:03:04    DEBUG:   cursor.fetchone:824 - Arrow BatchSize: 8
2020-03-26 18:03:04    DEBUG:   cursor.fetchone:824 - Arrow chunk info: batchCount 8, columnCount 8, use_numpy: 0
2020-03-26 18:03:04    DEBUG:   cursor.fetchone:824 - Current batch index: 0, rows in current batch: 48
2020-03-26 18:03:04    DEBUG:   cursor.fetchone:824 - Current batch index: 1, rows in current batch: 52
2020-03-26 18:03:04    DEBUG:   cursor.fetchone:824 - Current batch index

I think these come from some CPP code here. To mute these too, I also had to set snowflake.connector.CArrowIterator to a higher log level. I used code like this:

import logging
for name in logging.Logger.manager.loggerDict.keys():
    if 'snowflake' in name:
        logging.getLogger(name).setLevel(logging.WARNING)
        logging.getLogger(name).propagate = False
like image 27
hamx0r Avatar answered Oct 09 '22 23:10

hamx0r