4

I'm downloading TIGER data from the census site for the TIGER geocoder, but the computers IP has been blocked.

I'm trying the following now:

  1. Generate the loader script on the target machine.

  2. Copy that file into a different machine that is not banned.

  3. Keep only the wget statements in order to download the data.

  4. copy the downloaded data to the target machine.

  5. Edit the loader script commenting wget statements.

Is this a feasible approach?

Improve this question
7
  • 3
    If your computer's IP has been blocked/banned by the data custodian then attempting to do what you describe sounds like it may be illegal, or at least against the terms of service of that data custodian.
    – PolyGeo ♦
    Jul 21, 2017 at 1:57
  • 1
    @PolyGeo This is a reported issue with opensource data, is not a fail from my side: opendata.stackexchange.com/questions/10513/… , trac.osgeo.org/postgis/ticket/3698#comment:10
    – Luis Ramon Ramirez Rodriguez
    Jul 21, 2017 at 2:12
  • As far as I can see those two links do not show anything from the data custodian that suggests the block/ban by them was accidental. Consequently, I think you should be taking the issue of your access up with them.
    – PolyGeo ♦
    Jul 21, 2017 at 2:21
  • 3
    I kind of gave up myself trying to talk to census. Though feel free to try. Last I called a couple of months ago, they said can't help you it's on my end. I did hunt down that it's their akaimai setup doing it so it's hard to get ahold of anyone with a clue since it's so many levels up.
    – LR1234567
    Jul 21, 2017 at 3:02
  • 2
    As far as I can guess if you download more than a states worth of data, they start banning your ip and the ban could go on for days. Maybe they are being taxed too much and that's the way of limiting folks to keep bandwidth. Since I have several servers that need the data, I do as you described, download one state from each server and combine all so I can use the same data on any of the servers I need to load.
    – LR1234567
    Jul 21, 2017 at 18:41

3 Answers 3

Reset to default
4
+50

To simplify the download on multiple machines, you can use more advanced downloading tool (such as aria2c) that supports resume etc., and have a script continuously download the dataset from a file of links until it's done.

Quoting this SO answer, you can run the following script (loop_aria2.sh, under Linux) for a list of links list.txt:

#!/bin/bash
aria2c -j5 -i list.txt -c --save-session out.txt
has_error=`wc -l < out.txt`

while [ $has_error -gt 0 ]
do
  echo "still has $has_error errors, rerun aria2 to download ..."
  aria2c -j5 -i list.txt -c --save-session out.txt
  has_error=`wc -l < out.txt`
  sleep 10
done

You can partition the list.txt file for multiple machines.

If the ban is temporary (as the comments suggested), you also can adjust the sleep time above to wait for the ban to be lifted.

Improve this answer
3

I only see you mentioning http sites. Try using the Census' FTP site, ftp://ftp2.census.gov/geo/tiger/

Improve this answer
2
1

You could try using a VPN. If it is only the IP that has been blocked, you should be able to acces it through a VPN. There are loads of them.

Improve this answer
1
  • 3
    run a proxy through your script or run a VPN on your local machine. however this is a stack overflow question even though you want Geo Data your trying to remove an IP Ban.
    – NULL.Dude
    Jul 25, 2017 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.