0

The Census OnTheMap download links seems regular in formatting 1. This suggest someone may have already written an R package to download these files. Does such a package exist?

Other Comments:

If not, how might I do so? (I have a list of about 30 counties I wish to do it for). The links seem regular. Regular links, for SLC, Utah and Denver, Colorado:

https://onthemap.ces.census.gov/cgi-bin/report.py?report_id=otm_d2c5f4708fff45dd8a153aeea29b66ae&settings=%7B%22analysis_type%22%3A%22area_profile_report%22%2C%22view%22%3A%22summary%22%2C%22characteristic%22%3A%22c000%22%2C%22year%22%3A2015%2C%22job_type%22%3A%22jt00%22%2C%22ap_segment%22%3A%22s000%22%2C%22origin%22%3A%22work%22%2C%22color%22%3A%5B%22%230000AA%22%5D%7D&mode=export_geography&format=shp

https://onthemap.ces.census.gov/cgi-bin/report.py?report_id=otm_b874bc58baa84cbea0d57c2f654450ef&settings=%7B%22analysis_type%22%3A%22area_profile_report%22%2C%22view%22%3A%22summary%22%2C%22characteristic%22%3A%22c000%22%2C%22year%22%3A2015%2C%22job_type%22%3A%22jt00%22%2C%22ap_segment%22%3A%22s000%22%2C%22origin%22%3A%22work%22%2C%22color%22%3A%5B%22%230000AA%22%5D%7D&mode=export_geography&format=shp

The string for the download link consists of 3 parts, the first and third of which are identical, viz:

  • 1) https://onthemap.ces.census.gov/cgi-bin/report.py?report_id=otm_
  • 2a) d2c5f4708fff45dd8a153aeea29b66ae
  • 2b) b874bc58baa84cbea0d57c2f654450ef
  • 3) &settings=%7B%22analysis_type%22%3A%22area_profile_report%22%2C%22view%22%3A%22summary%22%2C%22characteristic%22%3A%22c000%22%2C%22year%22%3A2015%2C%22job_type%22%3A%22jt00%22%2C%22ap_segment%22%3A%22s000%22%2C%22origin%22%3A%22work%22%2C%22color%22%3A%5B%22%230000AA%22%5D%7D&mode=export_geography&format=shp

In other words, is there a way to use R to automate a query on a webmap server?

  • 1
    where have you looked already? – Spacedman Mar 6 '18 at 19:21
  • 1
    Why do you want a package for a single static download? If you want to download a file in R you can just use download.file(url(...)) but, in this case any type of URL scrubbing is unnecessary. If there are multiple files that you want to download then you need to quit giving us the single file download link! – Jeffrey Evans Mar 6 '18 at 19:23
  • 1
    Please do not just rephrase as a new question because you are not getting the response that you would like. I am voting to close because you could have easily modified your original question at: gis.stackexchange.com/questions/273670/… – Jeffrey Evans Mar 6 '18 at 19:26
  • The original question was different to this. As it was written in the there, it was not clear that there were meant to be multiple files, and hence the solution I provided was only for a single download link. This question is looking for a way to download multiple files. I advised him to include the URL as it gives an idea of how data could be downloaded. – Michael Harper Mar 6 '18 at 19:50
  • 1
    @MikeyHarper these are not static links, they are produced by a query on a webmap server, which is why the URL contains a "report_id" string. Unfortunately, there is nothing to scrape as the data is buried in the service and they did not provide an API for automated queries. – Jeffrey Evans Mar 6 '18 at 20:29
2

You do not need to query a webmap serve if you are looking for LEHD employment data. If you're searching for WAC, RAC, or OD data you can query the LED FTP directly or you can use a package I've designed. It is not yet on CRAN but here is the package link on github (https://github.com/jamgreen/lehdr). The package queries the LEHD FTP directly, so you can download preferred LED tables for any available year and state. Using this, you could download the data for your states of interest and then using a county file or matching block IDs you can subset.

If you don't want to use the package you can still download tables from the LODES FTP directly. This is the link to the latest version of the LEHD tables. From there you can write your own scripts if desired to download files directly from there.

  • Thank you, but I was looking for the points shapefile (geometry) not the tabular data. – Mox Mar 16 '18 at 21:16
  • What do you mean by points shapefile here? The LEHD provides the employment data and on the map simply displays it in different ways. There is no "point" LEHD data. That's a cartographic representation of the undelying employment data at the block level. If you want to reproduce a map like you see on OnTheMap, then that is a cartographic question and there are multiple options for you including dot density, proportional symbols or kernel density plots. – elmuertefurioso Mar 17 '18 at 15:45
  • So, this gets a bit awkward. Depending on which year of LEHD data your download, the points are in different locations. (Or were, last time I tested in 2011). They are also not the same as block centroids. I don't know why. – Mox Mar 19 '18 at 19:41
  • What were the points of, exactly? Block centroids? Employment? – elmuertefurioso Mar 19 '18 at 20:49
  • I don't know. I've not been able to find out. My working theory was that they were specially generated for reasons only the Census knows. – Mox Mar 20 '18 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.