How do I check whether Kivy UrlRequest Get is running on Android?

Question

I am trying to scrape a website using Kivy's UrlRequest. When I run the App from Android the code enters the UrlRequest function and never returns any information that it failed, an error occurred or success. How can I "listen" for log ouput to find out why nothing happens? Alternatively, what am I doing wrong with my UrlRequest code?

I have tried the following

1] Defined on_success, on_progress, on_error and on_failure functions to see if there is an output.

2] Created a settingspanel with different chunk_size, debug, decode and timeout values to see if it makes a difference when I change them.

3] Imported Kivy's LoggerHistory function to see if there is any error logs.

4] Looked at the buildozer android logcat output in the terminal to see what happens when I press the request button.

5] Changed url from https://www.google.com to http://www.google.com

6] Changed encoding from 'Latin-1' to 'utf-16' and back again.

7] Added a function using Urllib.request library to see if I can get anything and it gave me the following error:

07-30 21:51:09.933  9393  9417 I python  :  urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1051)>

UrlRequest Code

    def req_url(self, *args):
        url_chnk = self.the.config.getint('urlreqsettings', 'optionschunk')
        tme = self.the.config.getint('urlreqsettings', 'inttime')
        dcode = self.the.config.getboolean('urlreqsettings', 'booldecode')
        dbug = self.the.config.getboolean('urlreqsettings', 'booldebug')
        self.ids.result_success.color = (0,0,1,1)
        self.ids.result_success.text = "Busy..."
        # self.ids.conn_status.text = str(url_chnk)
        the_request = UrlRequest(self.the._url,on_progress=self.on_progress, on_success=self.on_success, on_error=self.on_error, on_failure=self.on_failure, chunk_size=url_chnk, timeout=tme, debug=dbug, decode=dcode)
        self.ids.req_info.text = str(the_request.is_finished) + " - " + str(the_request.chunk_size)
    def on_success(self, the_request, result, *args):
        self.ids.req_info.text = str(the_request.is_finished) + " - " + str(the_request.chunk_size)
        # print(the_request.resp_headers)
        if self.prog_circle:
            self.loader.dismiss()
            self.ids.result_success.color = (0,0,1,1)
            self.ids.result_success.text = "Successful UrlRequest!"
            self.ids.get_result.text = ""
            self.ids.get_result.text = str(result, encoding='Latin-1')
            self.prog_circle = False

My Url that I use

config.setdefaults('myurl', {
            'url': 'https://www.google.com'})

Ouput Logcat when on_release button calling Kivy's UrlRequest function

07-30 21:44:42.795  2122  2122 D NetworkController.WifiSignalController: Change in state from: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=true,activityOut=true,rssi=-51,lastModified=07-30 21:43:46,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null
07-30 21:44:42.795  2122  2122 D NetworkController.WifiSignalController:    to: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=false,activityOut=false,rssi=-51,lastModified=07-30 21:43:46,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null
07-30 21:44:42.861  1685  1748 E LightsService: Light requested not available on this device. 2
07-30 21:44:43.792  1685  1987 D WifiTrafficPoller: TRAFFIC_STATS_POLL true Token 625 num clients 1
07-30 21:44:43.794  1685  1987 D WifiTrafficPoller:  packet count Tx=76508 Rx=110820
07-30 21:44:43.797  2122  2402 D MessengerImpl: <send> message = 1 ,msg.sendingUid = 1000 ,Handler = Handler (com.android.systemui.statusbar.policy.WifiSignalController$WifiHandler) {3efccc8}
07-30 21:44:43.800  2122  2122 D NetworkController.WifiSignalController: Change in state from: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=false,activityOut=false,rssi=-51,lastModified=07-30 21:44:42,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null
07-30 21:44:43.800  2122  2122 D NetworkController.WifiSignalController:    to: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=false,activityOut=true,rssi=-51,lastModified=07-30 21:44:42,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null
07-30 21:44:44.797  1685  1987 D WifiTrafficPoller: TRAFFIC_STATS_POLL true Token 625 num clients 1
07-30 21:44:44.799  1685  1987 D WifiTrafficPoller:  packet count Tx=76509 Rx=110820
07-30 21:44:45.117   728   868 D SDM     : DisplayBase::BuildLayerStackStats: LayerStack layer_count: 7, app_layer_count: 6, gpu_target_index: 6, display type: 0
07-30 21:44:45.419   728   868 I chatty  : uid=1000(system) HwBinder:728_2 identical 1 line
07-30 21:44:45.637   728   868 D SDM     : DisplayBase::BuildLayerStackStats: LayerStack layer_count: 7, app_layer_count: 6, gpu_target_index: 6, display type: 0
07-30 21:44:45.800  1685  1987 D WifiTrafficPoller: TRAFFIC_STATS_POLL true Token 625 num clients 1
07-30 21:44:45.802  1685  1987 D WifiTrafficPoller:  packet count Tx=76526 Rx=110835
07-30 21:44:45.803  2122  2402 D MessengerImpl: <send> message = 1 ,msg.sendingUid = 1000 ,Handler = Handler (com.android.systemui.statusbar.policy.WifiSignalController$WifiHandler) {3efccc8}
07-30 21:44:45.805  2122  2122 D NetworkController.WifiSignalController: Change in state from: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=false,activityOut=true,rssi=-51,lastModified=07-30 21:44:43,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null
07-30 21:44:45.805  2122  2122 D NetworkController.WifiSignalController:    to: connected=true,enabled=true,level=4,inetCondition=1,iconGroup=IconGroup(Wi-Fi Icons),activityIn=true,activityOut=true,rssi=-51,lastModified=07-30 21:44:43,ssid="TP-Link_4837_5G",isTransient=false,statusLabel=null

Ouput Logcat when Urllib function button is released

07-30 21:51:09.902  9393  9417 I python  :  ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1051)
07-30 21:51:09.903  9393  9417 I python  :  
07-30 21:51:09.903  9393  9417 I python  :  During handling of the above exception, another exception occurred:
07-30 21:51:09.903  9393  9417 I python  :  
07-30 21:51:09.904  9393  9417 I python  :  Traceback (most recent call last):
07-30 21:51:09.904  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/app/main.py", line 499, in <module>
07-30 21:51:09.905  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/app.py", line 855, in run
07-30 21:51:09.906  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/base.py", line 504, in runTouchApp
07-30 21:51:09.906  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/core/window/window_sdl2.py", line 746, in mainloop
07-30 21:51:09.907  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/core/window/window_sdl2.py", line 478, in _mainloop
07-30 21:51:09.908  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/base.py", line 342, in idle
07-30 21:51:09.909  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/base.py", line 327, in dispatch_input
07-30 21:51:09.909  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/base.py", line 293, in post_dispatch_input
07-30 21:51:09.910  9393  9417 I python  :    File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
07-30 21:51:09.911  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/uix/behaviors/button.py", line 179, in on_touch_up
07-30 21:51:09.920  9393  9417 I python  :    File "kivy/_event.pyx", line 703, in kivy._event.EventDispatcher.dispatch
07-30 21:51:09.922  9393  9417 I python  :    File "kivy/_event.pyx", line 1214, in kivy._event.EventObservers.dispatch
07-30 21:51:09.923  9393  9417 I python  :    File "kivy/_event.pyx", line 1098, in kivy._event.EventObservers._dispatch
07-30 21:51:09.925  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/python-installs/urlsign/kivy/lang/builder.py", line 64, in custom_callback
07-30 21:51:09.926  9393  9417 I python  :    File "<string>", line 179, in <module>
07-30 21:51:09.926  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/app/main.py", line 300, in on_urllib_prog
07-30 21:51:09.927  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/app/main.py", line 303, in url_library
07-30 21:51:09.928  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 222, in urlopen
07-30 21:51:09.929  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 525, in open
07-30 21:51:09.929  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 543, in _open
07-30 21:51:09.930  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 503, in _call_chain
07-30 21:51:09.931  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 1360, in https_open
07-30 21:51:09.932  9393  9417 I python  :    File "/home/dataleaf/Build_Environ/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 1319, in do_open
07-30 21:51:09.933  9393  9417 I python  :  urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1051)>
07-30 21:51:09.933  9393  9417 I python  : Python for android ended.
07-30 21:51:10.229  1685  1987 D WifiTrafficPoller: TRAFFIC_STATS_POLL true Token 625 num clients 1
07-30 21:51:10.229  1685  1987 D WifiTrafficPoller:  packet count Tx=77858 Rx=112074
07-30 21:51:10.238  1685  1974 W InputDispatcher: channel '2fc1551 org.url_requesting.urlsign/org.kivy.android.PythonActivity (server)' ~ Consumer closed input channel or an error occurred.  events=0x9
07-30 21:51:10.238  1685  1974 E InputDispatcher: channel '2fc1551 org.url_requesting.urlsign/org.kivy.android.PythonActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
07-30 21:51:10.239  1685  4609 I ActivityManager: Process org.url_requesting.urlsign (pid 9393) has died: fore TOP 
07-30 21:51:10.240  1685  2993 I WindowManager: WIN DEATH: Window{2fc1551 u0 org.url_requesting.urlsign/org.kivy.android.PythonActivity}
07-30 21:51:10.240  1685  2993 W InputDispatcher: Attempted to unregister already unregistered input channel '2fc1551 org.url_requesting.urlsign/org.kivy.android.PythonActivity (server)'
07-30 21:51:10.240   648   648 I Zygote  : Process 9393 exited cleanly (255)
07-30 21:51:10.241  1685  1750 W libprocessgroup: kill(-9393, 9) failed: No such process
07-30 21:51:10.241  1685  1750 I libprocessgroup: Successfully killed process cgroup uid 10272 pid 9393 in 0ms
07-30 21:51:10.243  1685  4609 W ActivityManager: Force removing ActivityRecord{e32495c u0 org.url_requesting.urlsign/org.kivy.android.PythonActivity t6783}: app died, no saved state

Update

I removed the "s" from my url "https://orob.co.za" again, uninstalled the App completely from my phone and installed again (not updated). This time the Urllib function gave the following error:

07-30 22:15:10.961 10706 10730 I python  :  socket.timeout: timed out
07-30 22:15:10.961 10706 10730 I python  : Python for android ended.

While the Kivy UrlRequest function for the first time entered the on_progress function. But the request did not succeed, only times out.

Update on update and success albeit not HTTP... S!

Success! I cracked it. When I had tried [5] above I had only updated the App on my phone and not uninstalled it. I had also lost the plot and changed the domain to a different one than the one I used in the initial tests, which meant I got the right results but the wrong facts. I have changed it back to "http://www.google.com" and the App returned the scrape ("Latin-1") for UrlRequest and Urllib. Very exiting!

The problem seems to be the SSL certificate. I have openssl as part of my buidozer.spec file, so why is this not working in Android with HTTPS!

Update on update on update...

So my previous finding is correct. Certificate verification is failing on the Kivy Urlrequest function for "HTTPS://..." url also. After tweeking my code I found that the on_error function's error output is a string and not a byte-like object, that meant I tried decoding a string which caused the error and blocked the error output in my App:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)

Question edited

Can someone please help on how I can return the result from the scrape.. from url with https?

Answer 1

I am answering my own question as I have finally solved it and I am not sure of another way to inform others.
The SSL certificate question has many answers on this forum, but the information that solved my question are mostly comments.

Solution

Using Python 3.6.8 and Kivy 1.11.0

Step 1> I had to pip3 install requests to get the certifi python library.
See @Petar Luktina comments "HTTPS Request in Kivy"

Step 2> Add the ca_file=certifi.where() parameter to your UrlRequest(). As an extra I added the verify=True parameter.
See @Erik comments "Kivy UrlRequest with https"

Step 3> Add certifi and openssl to buildozer.spec file as Application requirements, else you would get a ModuleNotFoundError: No module named '_ctypes'. The error is due to an explicitly imported library that is not declared in the .spec file or does not form part of an already declared package

Step 4> Add functions to handle on_success parameter (not necessary but it worked for me).

My coded answer:

import certifi as cfi
from kivy.network.urlrequest import UrlRequest

    def req_url(self, *args):
        the_request = UrlRequest(self.the._url, on_success=self.on_success, 
                               on_error=self.on_error, on_failure=self.on_failure,
                                     ca_file=cfi.where(), verify=True)

Presto! It now works on Android.