3

I'm downloading TIGER data from the census site for the TIGER geocoder, but the computers IP has been blocked.

I'm trying the following now:

  1. Generate the loader script on the target machine.

  2. Copy that file into a different machine that is not banned.

  3. Keep only the wget statements in order to download the data.

  4. copy the downloaded data to the target machine.

  5. Edit the loader script commenting wget statements.

Is this a feasible approach?

  • 3
    If your computer's IP has been blocked/banned by the data custodian then attempting to do what you describe sounds like it may be illegal, or at least against the terms of service of that data custodian. – PolyGeo Jul 21 '17 at 1:57
  • @PolyGeo This is a reported issue with opensource data, is not a fail from my side: opendata.stackexchange.com/questions/10513/… , trac.osgeo.org/postgis/ticket/3698#comment:10 – Luis Ramon Ramirez Rodriguez Jul 21 '17 at 2:12
  • As far as I can see those two links do not show anything from the data custodian that suggests the block/ban by them was accidental. Consequently, I think you should be taking the issue of your access up with them. – PolyGeo Jul 21 '17 at 2:21
  • 2
    I kind of gave up myself trying to talk to census. Though feel free to try. Last I called a couple of months ago, they said can't help you it's on my end. I did hunt down that it's their akaimai setup doing it so it's hard to get ahold of anyone with a clue since it's so many levels up. – LR1234567 Jul 21 '17 at 3:02
  • 2
    As far as I can guess if you download more than a states worth of data, they start banning your ip and the ban could go on for days. Maybe they are being taxed too much and that's the way of limiting folks to keep bandwidth. Since I have several servers that need the data, I do as you described, download one state from each server and combine all so I can use the same data on any of the servers I need to load. – LR1234567 Jul 21 '17 at 18:41
4
+50

To simplify the download on multiple machines, you can use more advanced downloading tool (such as aria2c) that supports resume etc., and have a script continuously download the dataset from a file of links until it's done.

Quoting this SO answer, you can run the following script (loop_aria2.sh, under Linux) for a list of links list.txt:

#!/bin/bash
aria2c -j5 -i list.txt -c --save-session out.txt
has_error=`wc -l < out.txt`

while [ $has_error -gt 0 ]
do
  echo "still has $has_error errors, rerun aria2 to download ..."
  aria2c -j5 -i list.txt -c --save-session out.txt
  has_error=`wc -l < out.txt`
  sleep 10
done

You can partition the list.txt file for multiple machines.

If the ban is temporary (as the comments suggested), you also can adjust the sleep time above to wait for the ban to be lifted.

3

I only see you mentioning http sites. Try using the Census' FTP site, ftp://ftp2.census.gov/geo/tiger/

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

1

You could try using a VPN. If it is only the IP that has been blocked, you should be able to acces it through a VPN. There are loads of them.

  • 3
    run a proxy through your script or run a VPN on your local machine. however this is a stack overflow question even though you want Geo Data your trying to remove an IP Ban. – NULL.Dude Jul 25 '17 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.