13

I am working on a plugin, where sending requests to an external server is part of the process. When using no proxy (like from my personal computer), this works just fine:

import urllib

# example url from an publicly accessible OTP-Server
url = 'https://api.digitransit.fi/routing/v1/routers/hsl/isochrone?fromPlace=60.169,24.938&mode=WALK,TRANSIT&date=2019-11-01&time=08:00:00&maxWalkDistance=500&cutoffSec=1800&cutoffSec=3600'
headers = {"accept":"application/x-zip-compressed"}
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request)

with open('D://isochrones.zip', 'wb') as f:
    f.write(response.read())

However, I have now tried this from my company's network where we are using a proxy. It is set up within QGIS settings and works just fine to browse the plugin repository or use external basemaps. But my plugin is not working any longer.

So I tried to set a proxy for urllib via reading the QGIS proxy settings from QSettings() as:

# Reading QgsProxySettings via QSettings().value("proxy/proxyEnabled", "")
# Concatenating the settings as proxyUser + ':' + proxyPassword + '@' + proxyHost + ':' + proxyPort
# And using it as
proxy_support = urllib.request.ProxyHandler(proxyhandledict)
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)

before doing the request. But this does not work properly and as mentioned by QGIS developers, using external librarys should be avoided and instead using QgsNetworkAccessManager is encouraged.

So I was trying this (rather old) solution and some snippets like

# example url from an publicly accessible OTP-Server
url = 'https://api.digitransit.fi/routing/v1/routers/hsl/isochrone?fromPlace=60.169,24.938&mode=WALK,TRANSIT&date=2019-11-01&time=08:00:00&maxWalkDistance=500&cutoffSec=1800&cutoffSec=3600'
url = QUrl(url)

networkAccessManager = QgsNetworkAccessManager.instance()
request = QNetworkRequest(url)
request.setHeader(QNetworkRequest.ContentTypeHeader,"application/x-zip-compressed")
request.setRawHeader(b"Accept", b"application/x-zip-compressed")
response = networkAccessManager.get(request)

with open('D://isochrones.zip', 'wb') as f:
    f.write(response.readAll())

but I never get a valid response zipfile from the OpenTripPlanner server, but only the json repsonse, which means that the header was missing or not sent. So I really do need to send the headers to get my desired response zipfile.

How can I send requests including headers by using the QGIS 3 proxy settings and save its reply to a file?

3
  • 1
    Look at this one : stackoverflow.com/a/3199078 Aug 25 '20 at 14:21
  • Could you get a valid response on your exact same request+headers when using other tools (wget, curl, or whatever)? Sep 26 '21 at 10:48
  • @swiss_knight I did only try requests library which works fine, but only if requests is available, thats why I use urllib for now.
    – MrXsquared
    Sep 26 '21 at 11:13
2
+100

Using the QgsNetworkAccessManager you can do something like this:

url = "your.url.xyz" # a string            
headers = {b"Accept":b"application/x-zip-compressed"} # a dictionary
s = QgsSettings()
proxy = QNetworkProxy()
proxyEnabled = s.value("proxy/proxyEnabled", "")
proxyType = s.value("proxy/proxyType", "" )
proxyHost = s.value("proxy/proxyHost", "" )
proxyPort = s.value("proxy/proxyPort", "" )
proxyUser = s.value("proxy/proxyUser", "" )
proxyPassword = s.value("proxy/proxyPassword", "" )
QNM = QgsNetworkAccessManager()
QNM.setTimeout(20000) # if needed
if proxyEnabled == "true":
    proxy.setType(QNetworkProxy.HttpProxy)
    proxy.setHostName(proxyHost)
    if proxyPort != "":
        proxy.setPort(int(proxyPort))
    proxy.setUser(proxyUser)
    proxy.setPassword(proxyPassword)
    QNetworkProxy.setApplicationProxy(proxy)
    QNM.setupDefaultProxyAndCache()
    QNM.setFallbackProxyAndExcludes(proxy,[""],[""])
request = QNetworkRequest(QUrl(url))
if headers:
    for header in headers.keys():
        request.setRawHeader(header, headers[header])
reply = QNM.blockingGet(request)
reply.content() # this should be your compressed returned zip
reply.content().data() # for json

Sending the headers this way should work, I used it on the Israeli open data plugin.
You can download the plugin here and check out the postWithProxy function (line 347), getWithProxy is what I copied here but doesn't actually use headers, though they are written the same way.

I know for sure that headers are being sent this way because I wrote the server side which checks them so this should at least point you in the right direction.

3
  • Thanks, this already helps a lot. Working fine on my personal machine with 3.20, where I cant test with a working proxy set up. On my companies machine it does not work with a proxy set up, but most likely due to the very old 3.4.3 version of QGIS I am limited to, where after removing all by this version unsupported stuff, I only get empty responses. Not sure if due to having to replace .blockingGet() with .get() or reply.content() with reply.readAll() or the proxy itself. Which QGIS version do you use or which one ist the oldest you have tested this with?
    – MrXsquared
    Sep 30 '21 at 12:50
  • Not sure which was the oldest we tested with, must have been 3.6 or newer, from what I can see 3.6 is when blockingGet was added. From what it looks like reply.content() in 3.6 and up should return the same type as reply.readAll(), I can't see any documentation for .get() that might be your issue with the 3.4 instances. have you tried using createRequest?
    – Dror Bogin
    Oct 2 '21 at 10:20
  • Note that using createRequest uses QT objects and not Qgs objects, for example the request is built using qnetworkrequest
    – Dror Bogin
    Oct 2 '21 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.