Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can InsecureRequestWarnings be disabled in the requests module?

This post explains how to disable the warnings in urllib3, but I'm using a session object with the requests module. The warnings are occurring because the verify (ssl) parameter is set to False in order to access a page.

e.g.

url = "https://www.booking.com/searchresults.en-gb.html"

querystring = {"label":"gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmcgV1c19vcogBAZgBMbgBB8gBDdgBA-gBAfgBApICAXmoAgM","lang":"en-gb","sid":"5f9b0b3af27a0a0b48017c6c387d8224","track_lsso":"2","sb":"1","src":"searchresults","src_elem":"sb","error_url":"https://www.booking.com/searchresults.en-gb.html?label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmcgV1c19vcogBAZgBMbgBB8gBDdgBA-gBAfgBApICAXmoAgM;sid=5f9b0b3af27a0a0b48017c6c387d8224;class_interval=1;dest_id=30;dest_type=country;dtdisc=0;group_adults=1;group_children=0;inac=0;index_postcard=0;label_click=undef;mih=0;no_rooms=1;offset=0;postcard=0;raw_dest_type=country;room1=A;sb_price_type=total;src=searchresults;src_elem=sb;ss=" + country + ";ss_all=0;ssb=empty;sshis=0;ssne=" + country + ";ssne_untouched=" + country + "&;","ss":country,"ssne":country,"ssne_untouched":country,"dest_id":"30","dest_type":"country","checkin_monthday":"","checkin_month":"","checkin_year":"","checkout_monthday":"","checkout_month":"","checkout_year":"","room1":"A","no_rooms":"1","group_adults":"1","group_children":"0"}

headers = {
        'upgrade-insecure-requests': "1",
        'user-agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36",
        'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        'referer': "https://www.booking.com/searchresults.en-gb.html?label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmcgV1c19vcogBAZgBMbgBB8gBDdgBA-gBAfgBApICAXmoAgM&lang=en-gb&sid=5f9b0b3af27a0a0b48017c6c387d8224&track_lsso=2&sb=1&src=searchresults&src_elem=sb&error_url=https%3A%2F%2Fwww.booking.com%2Fsearchresults.en-gb.html%3Flabel%3Dgen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmcgV1c19vcogBAZgBMbgBB8gBDdgBA-gBAfgBApICAXmoAgM%3Bsid%3D5f9b0b3af27a0a0b48017c6c387d8224%3Bclass_interval%3D1%3Bdest_id%3D30%3Bdest_type%3Dcountry%3Bgroup_adults%3D1%3Bgroup_children%3D0%3Blabel_click%3Dundef%3Bmih%3D0%3Bno_rooms%3D1%3Boffset%3D0%3Braw_dest_type%3Dcountry%3Broom1%3DA%3Bsb_price_type%3Dtotal%3Bsrc%3Dsearchresults%3Bsrc_elem%3Dsb%3Bss%3D" + country + "%3Bssb%3Dempty%3Bssne%3D" + country + "%3Bssne_untouched%3D" + country + "%26%3B&ss=" + country + "&ssne=" + country + "&ssne_untouched=" + country + "&dest_id=30&dest_type=country&checkin_monthday=&checkin_month=&checkin_year=&checkout_monthday=&checkout_month=&checkout_year=&room1=A&no_rooms=1&group_adults=1&group_children=0",
        'content-Encoding': "br",
        'accept-language': "en-US,en;q=0.8",
        'content-type': "text/html;charset=UTF-8",
        'cookie': "zz_cook_tms_seg1=1; zz_cook_tms_seg3=7; zz_cook_tms_ed=1; zz_cook_tms_hlist=1231360; _ga=GA1.2.1568457147.1491521761; BJS=-; utag_main=v_id:015b45a012b10048b0bb15efec880407900ef07100838$_sn:1$_ss:0$_st:1491528314624$ses_id:1491521770165%3Bexp-session$_pn:10%3Bexp-session$4split:2; vpmss=1; zz_cook_tms_seg1=1; zz_cook_tms_seg3=7; has_preloaded=1; zz_cook_tms_ed=1; _gat=1; zz_cook_tms_hlist=1231360; _ga=GA1.2.1568457147.1491521761; BJS=-; utag_main=v_id:015b45a012b10048b0bb15efec880407900ef07100838$_sn:1$_ss:0$_st:1491528297719$ses_id:1491521770165%3Bexp-session$_pn:9%3Bexp-session$4split:2; _tq_id.TV-451827-1.3b4c=b2a66dddb07863c1.1491521784.0.1491526498..; bkng=11UmFuZG9tSVYkc2RlIyh9YWJdm48m5cJDGAKDgsOMdIf7GAOd2XfnSWwDn3wa4gpryaHb0j9bqCgIooHJpCTb5ICfHL52QknfZGWBSrbMuuvwazeTr5lQsnZcUdV2EQsQz3i0Nf7uaTZSaVW9fbHlcCNrYswEwb5XFP6SFj4WWPH6u7mu2jGbvUbUrHP0MnfQN6lT8uLH95I%3D; lastSeen=0; bkng=11UmFuZG9tSVYkc2RlIyh9YWJdm48m5cJD4P2Fap9agWiA2cAEcL9fvsxvcdnfnCC9y3516kQSx8qK7PLf9mmmBBZChjNiFTEA6XmVGUtjLGiYQ281ua2dfQcX87SBuBWY%2F1vf6ATBfLAesm8JbemGaO%2BDHB7Mf8wAJIgnnzt3ZhrlLKc8OvkHVtANpfO6HfvirPd9hUxPTO4%3D; lastSeen=0",
        'cache-control': "no-cache",
        'postman-token': "124b1e3b-c4de-9ab0-162f-003770797f9f"
        }

response = BeautifulSoup(session.request("GET", url, headers=headers, params=querystring, verify=False).content, "html.parser")
like image 654
Phillip Avatar asked Feb 04 '23 17:02

Phillip


1 Answers

SSL, security and other warnings can be disabled with urllib3.disable_warnings() (requests uses urllib3 internally). Specifically, SSL verification warnings can be suppressed if we use disable_warnings() with an InsecureRequestWarning object.

import requests
from requests.packages import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

url = "https://www.booking.com/searchresults.en-gb.html"
with requests.Session() as s:
    r = s.get(url, verify=False)
    html = r.text

Note that disabling SSL Peer Verification (verify=False) may be a security issue as it can lead to MITM attacks.

like image 117
t.m.adam Avatar answered Feb 07 '23 19:02

t.m.adam