Sending requests including headers via PyQGIS by using QGIS Proxysettings

Question

I am working on a plugin, where sending requests to an external server is part of the process. When using no proxy (like from my personal computer), this works just fine:

import urllib

# example url from an publicly accessible OTP-Server
url = 'https://api.digitransit.fi/routing/v1/routers/hsl/isochrone?fromPlace=60.169,24.938&mode=WALK,TRANSIT&date=2019-11-01&time=08:00:00&maxWalkDistance=500&cutoffSec=1800&cutoffSec=3600'
headers = {"accept":"application/x-zip-compressed"}
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request)

with open('D://isochrones.zip', 'wb') as f:
    f.write(response.read())

However, I have now tried this from my company's network where we are using a proxy. It is set up within QGIS settings and works just fine to browse the plugin repository or use external basemaps. But my plugin is not working any longer.

So I tried to set a proxy for urllib via reading the QGIS proxy settings from QSettings() as:

# Reading QgsProxySettings via QSettings().value("proxy/proxyEnabled", "")
# Concatenating the settings as proxyUser + ':' + proxyPassword + '@' + proxyHost + ':' + proxyPort
# And using it as
proxy_support = urllib.request.ProxyHandler(proxyhandledict)
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)

before doing the request. But this does not work properly and as mentioned by QGIS developers, using external librarys should be avoided and instead using QgsNetworkAccessManager is encouraged.

So I was trying this (rather old) solution and some snippets like

# example url from an publicly accessible OTP-Server
url = 'https://api.digitransit.fi/routing/v1/routers/hsl/isochrone?fromPlace=60.169,24.938&mode=WALK,TRANSIT&date=2019-11-01&time=08:00:00&maxWalkDistance=500&cutoffSec=1800&cutoffSec=3600'
url = QUrl(url)

networkAccessManager = QgsNetworkAccessManager.instance()
request = QNetworkRequest(url)
request.setHeader(QNetworkRequest.ContentTypeHeader,"application/x-zip-compressed")
request.setRawHeader(b"Accept", b"application/x-zip-compressed")
response = networkAccessManager.get(request)

with open('D://isochrones.zip', 'wb') as f:
    f.write(response.readAll())

but I never get a valid response zipfile from the OpenTripPlanner server, but only the json repsonse, which means that the header was missing or not sent. So I really do need to send the headers to get my desired response zipfile.

How can I send requests including headers by using the QGIS 3 proxy settings and save its reply to a file?

Answer 1

Using the QgsNetworkAccessManager you can do something like this:

url = "your.url.xyz" # a string            
headers = {b"Accept":b"application/x-zip-compressed"} # a dictionary
s = QgsSettings()
proxy = QNetworkProxy()
proxyEnabled = s.value("proxy/proxyEnabled", "")
proxyType = s.value("proxy/proxyType", "" )
proxyHost = s.value("proxy/proxyHost", "" )
proxyPort = s.value("proxy/proxyPort", "" )
proxyUser = s.value("proxy/proxyUser", "" )
proxyPassword = s.value("proxy/proxyPassword", "" )
QNM = QgsNetworkAccessManager()
QNM.setTimeout(20000) # if needed
if proxyEnabled == "true":
    proxy.setType(QNetworkProxy.HttpProxy)
    proxy.setHostName(proxyHost)
    if proxyPort != "":
        proxy.setPort(int(proxyPort))
    proxy.setUser(proxyUser)
    proxy.setPassword(proxyPassword)
    QNetworkProxy.setApplicationProxy(proxy)
    QNM.setupDefaultProxyAndCache()
    QNM.setFallbackProxyAndExcludes(proxy,[""],[""])
request = QNetworkRequest(QUrl(url))
if headers:
    for header in headers.keys():
        request.setRawHeader(header, headers[header])
reply = QNM.blockingGet(request)
reply.content() # this should be your compressed returned zip
reply.content().data() # for json

Sending the headers this way should work, I used it on the Israeli open data plugin.
You can download the plugin here and check out the postWithProxy function (line 347), getWithProxy is what I copied here but doesn't actually use headers, though they are written the same way.

I know for sure that headers are being sent this way because I wrote the server side which checks them so this should at least point you in the right direction.