-1

I'd like to automate downloading point shapefile data from Census OnTheMap. There is a link which can be used to download the data, as provided below:

https://onthemap.ces.census.gov/cgi-bin/report.py?report_id=otm_d2c5f4708fff45dd8a153aeea29b66ae&settings=%7B%22analysis_type%22%3A%22area_profile_report%22%2C%22view%22%3A%22summary%22%2C%22characteristic%22%3A%22c000%22%2C%22year%22%3A2015%2C%22job_type%22%3A%22jt00%22%2C%22ap_segment%22%3A%22s000%22%2C%22origin%22%3A%22work%22%2C%22color%22%3A%5B%22%230000AA%22%5D%7D&mode=export_geography&format=shp

Is it possible to directly download this within R?

Other Comments:

  • I don't want Decennial Census, or ACS (or else I'd use totalcensus or tidycensus).
  • I already have the tabular data (CSV).
  • Whilst this question is about using Census On the Map, is there an R package to do so?
  • 3
    OnTheMap is a webmap service with internal tools that build a query and then post the data in a zip format. There is no API provided by the service to do this outside the map interface so, it is very difficult to construct and scrub results. Their "text only" tools are nothing but a search engine for the site and do not provide an API either. Sorry, but it looks like you will have to construct the downloads using the map interface and then access the static download. It honestly does not look like that much work to do manually. – Jeffrey Evans Mar 6 '18 at 19:33
  • Official word from Census was: "Unfortunately there isn't an easy way to make requests for OnTheMap analysis results at this time. The original application was designed a decade or so ago and the underlying data API design is tightly tied to the web app. This was a time before open APIs were as prevalent as they are today. There isn't an easy way to get under the hood currently. There have been some users who've been successful but it involves being familiar with web browser developer tools and learning how to design a program to mimic the web app." – Mox Mar 7 '18 at 20:49
1

You can download and unzip the file within R. The main thing to make this work though is that you need to make sure the download is in binary mode as highlighted in this post.

The script below downloads the file, unzips it into a directory called "shp", then deletes the zip file.

# Specify link
downloadLink <- "https://onthemap.ces.census.gov/cgi-bin/report.py?report_id=otm_d2c5f4708fff45dd8a153aeea29b66ae&settings=%7B%22analysis_type%22%3A%22area_profile_report%22%2C%22view%22%3A%22summary%22%2C%22characteristic%22%3A%22c000%22%2C%22year%22%3A2015%2C%22job_type%22%3A%22jt00%22%2C%22ap_segment%22%3A%22s000%22%2C%22origin%22%3A%22work%22%2C%22color%22%3A%5B%22%230000AA%22%5D%7D&mode=export_geography&format=shp"

# Download file
download.file(url = downloadLink, 
              destfile = "download.zip", mode="wb")
unzip(zipfile = "download.zip", exdir = "shp")
file.remove("download.zip")

To show it has worked, here is a quick plot:

# Plot results
library(sf)
a <- sf::read_sf("shp/points_2015.shp")
plot(a)

enter image description here

Warning: I ran this on Windows. I don't think this should be different on other operating systems but commands like these can have a habit of being different between platforms.

  • Sorry, I was a bit unclear in my question. I wanted to either find an R package that would do the automation, or some way to parse the string of the otm download link in such a way that I could write a FOR loop in R to download all the files. – Mox Mar 6 '18 at 18:04
  • It might be worth starting a different question then instead of altering htis one. The original question seemed to be about downloading this zip file (which I hope this answered) – Michael Harper Mar 6 '18 at 18:22
  • Also, removing the URL from the question removes a vital bit of information. It seems like the only way to achieve this automatically is to do some form of web scraping, and having the URL structure is vital for this. – Michael Harper Mar 6 '18 at 18:24
  • 1
    Ok, will do that. – Mox Mar 6 '18 at 18:36
  • Official word from census is that the API is not exposed, so using a URL as a link will not work. – Mox Mar 7 '18 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.