Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I've contacted the author of this NYC homicide map from the NY Times several times to see if he could send me a spreadsheet of information the data is based off of, but I have not had any response (for a very long time).

Is there anyway I could scrap the data from the google map directly? I downloaded the swf file and converted to xml, but even within there I can not tell where the information is I want. Here is the zipped up file of the converted xml.

I'm hopeing someone with more knowledge of online mapping can quickly tell me if this possible or not (or where I should be looking in the xml file).

share|improve this question
    
Wow, interesting to see how it all works. What about the underlying data from cottonmap.com.au/Map.aspx ? I suspect this has something but stumped as to what to do with it? maps.agrecon.com/map/mapserv.exe?map=F:/MapserverData/… –  Willy Mar 7 '12 at 11:18
add comment

4 Answers

up vote 11 down vote accepted

Quickly glancing at the page in Firebug and looking at the network calls, you can see where they are pulling the data from. Seems to be a couple of XML files, namely:

http://graphics8.nytimes.com/packages/xml/map_feed_victims.txt?c=2182

and

http://graphics8.nytimes.com/packages/xml/map_feed_incidents.txt?c=2182

share|improve this answer
1  
Very cool, thank you. If you feel so inclined, I would appreciate alittle more hand holding about how you figured such information out. –  Andy W Mar 5 '12 at 13:35
3  
Basically, if you have Firefox, get the Firebug add-on. Visit the page and open Firebug. Refresh and look at the "Network" tab and you will see all of the calls the page is making, including the links above. –  ericoneal Mar 5 '12 at 14:31
    
I just would like to remind you that this file appears to have people's names in it and therefore their privacy is being violated? –  George Mar 6 '12 at 12:21
    
@George, the data is publicly available within the google map already listed on the Times Web-page (tool-tips provide all of the information). It just isn't in a format that allows me to readily manipulate the data otherwise. If your concerned about privacy your beef is with the NY Times. –  Andy W Mar 6 '12 at 13:27
    
@AndyW you are correct, it is with them. It seems wrong to have their names publicized. They should remove that from their website (at least the names, in the json format). –  George Mar 8 '12 at 17:16
add comment

I don't know if you can get the exact same data from the NYC Open data repository, but here is a link.


A slightly different approach could be to try to gather the data using the New York Times API: http://prototype.nytimes.com/gst/apitool/index.html

share|improve this answer
    
It is not available from the NYC Open data repository. I believe it is culled from NY Times articles themselves on reported homicides. The NYPD does not release any dataset about homicides to that level of specificity. –  Andy W Mar 4 '12 at 20:59
    
They do have a "geographic" API in that list, but no documentation for it (I assume not up and running?) May be worth leaving the answer up just for that info about the Times API, it would be very cool if eventually they have a full semantic GIS api (although that is just pure speculation). –  Andy W Mar 4 '12 at 21:19
add comment

+1 to @ericoneal's answer, but for the sake of noting an alternative approach, you could also download and install Fiddler. Fiddler routes your port-80 traffic through a proxy and provides you an interface for poking-around in the HTTP responses that follow your web request.

I'll describe the usage. In the screenshot, I just launched Fiddler, then opened your link in IE. All the data starts streaming-in without my doing anything else. Once it's settled, at left, I clicked on one of the returns (map_feed_incidents.txt, as noted by Eric), then at top-right, I select Inspectors. The pane at bottom-right provides several inspection formats. I tried a few, and the screen shows the TextView.

At a glance, the content appears to be line-break and tab-delimited (it's definitely not real XML). The top line specifies the file format, and every other line is an incident record. Here's the top-line and first record from the _incidents file (scroll right and note the id field):

LAT:DOUBLE  LONG:DOUBLE incident_date:STRING    incident_time:STRING    boro:STRING num_victims:INTEGER primary_motive:STRING   id:INTEGER  weapon:STRING   light_dark:STRING   year:INTEGER
40.665626   -73.909699  01/01/08    02:09   Brooklyn    1       7   gun D   2008

The lat/long is obvious. The other two files (_victims and _perpetrators) use the same approach. Here's the top line and first record from the _perps table:

incident_id:INTEGER sex:STRING  race:STRING age:INTEGER
7   M   B   20

The presence of incident_id is useful. Both _victims and _perps have this column, and it relates their data back to the geo-tagged _incidents table using that table's id column.

enter image description here

As an aside.. I have to agree with George and wonder why they included the victim's name. That seems like a major ethical oversight. While it's meaningless as a mapped attribute, I would not be surprised to see the perpetrator's name. But the victim's? At first I thought this may have been an unused element in the data payload, but it's really in the map?!?! That's a very questionable decision, and it leads me to believe nobody is using that map. Otherwise I think some criticism would've emerged from the general public.

share|improve this answer
add comment

How about not doing this? Are you sure that using their data for your purposes is allowed by New York Times? Or are you convinced that the data is not copyrightable?

I quickly glanced at the source and the bottom of the page lists it as Copyright to the New York Times.

If you did this to my site then I would be extremely angry.

Update 2012-03-10 The questioner claims that this is legal for two reasons: that the information is data and not subject to copyright; and that they wish to use this for research purposes and thus copyright does not apply to them.

Copyrightability of data

US case law is a little different to many other countries in how far it goes in defining data as not being copyrightable. The leading case in the area is Feist v. Rural. The subject matter was Rural refusing to license its telephone listings to Feist so Feist used them anyway. Rural sued for copyright infringement and lost. Quoting from the case:

The constitutional requirement necessitates independent creation plus a modicum of creativity. Since facts do not owe their origin to an act of authorship, they are not original, and thus are not copyrightable. Although a compilation of facts may possess the requisite originality because the author typically chooses which facts to include, in what order to place them, and how to arrange the data so that readers may use them effectively, copyright protection extends only to those components of the work that are original to the author, not to the facts themselves. This fact/expression dichotomy severely limits the scope of protection in fact-based works.

Canadian law is not relevant to the current question but I'll raise it to compare and contrast. The leading case here is CCH Canadian Ltd vs Law Society of Upper Canada. Quoting Wikipedia's quoting of the judgment:

McLachlin rejects Justice O'Connor's "minimal degree of creativity" test but agrees with her assessment of the "sweat of the brow" approach and finds it too low a requirement. Instead, McLachlin takes the middle ground by requiring "that an original work be the product of an exercise of skill and judgment" where "skill" is "the use of one's knowledge, developed aptitude or practised ability in producing the work" and "judgment" is "the use of one's capacity for discernment or ability to form an opinion or evaluation by comparing different possible options in producing the work". (para 16) As well, "[t]he exercise of skill and judgment required to produce the work must not be so trivial that it could be characterized as a purely mechanical exercise." (para 16) Importantly, it is required that the work "must be more than a mere copy of another work." (para. 16) However, "creativity is not required to make a work 'original'." (para. 25)

Use of copyrighted works for research purposes

In copyright law, fair use (US) or fair dealing (common law countries) recognizes that a work shouldn't exist all by itself but becomes part of the culture by being discussed, criticized, referenced, etc.

In US law, "research" is specifically allowed as a reason to for fair use but note that one of the four factors is "the amount and substantiality of the portion used in relation to the copyrighted work as a whole".

One consideration is whether the new work adds something different than the old. Quoting Wikipedia again, "To justify the use as fair, one must demonstrate how it either advances knowledge or the progress of the arts through the addition of something new. A key consideration is the extent to which the use is interpreted as transformative, as opposed to merely derivative."

Indeed, my preparation of this extended answer with lots of quotes comes under fair use (in my opinion). I'm welding a bunch of information together to create a new work.

Legality of site scraping

There was a recent case in the BC Supreme Court, Century 21 v Rogers, which discussed similar issues to what you propose. Quoting Michael Geist's comments,

Century 21's terms prohibited copying or scraping its content. By doing so, Zoocasa breached the contract...copying the full length property descriptions for several months constituted copyright infringement copying truncated, shorter descriptions did not (an example the court provides is "212-819 Hamilton St, Vancouver, B.C. V6Z 6M2 $285,000 - 1 Bed, 1 Bath - Great Newly Updated Jr 1 Bedroom at 819 Hamilton. This is a great West facing 1 bedroom suite which has lots of great. …") copying photographs of properties constituted copyright infringement

Certainly, being a Canadian case (and only a provincial one so far) it's not very persuasive in a US court but I would suggest that a US court probably isn't going to judge things very differently.

Conclusion Purely in my opinion, and I'm certainly not a lawyer and this is not legal advice, I'd suggest that scraping the data in order to produce a very similar version of the same map, especially for commercial purposes, would probably be copyright infringement.

Using the data for research purposes to create a new work, especially if you don't use all of the data, is probably safe under both the fair use and data doctrines.

share|improve this answer
    
I'm not quite sure why your chastising me. If you publicly publish your data, what do you expect people to do with it? Personally I wouldn't use it for in any direct profitable way (I'm a researcher), and utilizing such data for research purposes wouldn't be any different than using information published in the newspaper. –  Andy W Mar 8 '12 at 18:41
    
This isn't an answer to my question either. Perhaps bring it up on meta (or another question on the main site) if it really bothers you and you would like other community input. –  Andy W Mar 8 '12 at 18:43
    
Hmm, I'll admit I'm not a lawyer, and so I could be confused about what is and what is not "copyrightable", but this page suggests that data can not be copyrighted. The Time's does have some guidelines about re-distributing their work, so redistributing the print or web content verbatim would be in some violation I would imagine, but the data itself is not subject to such restrictions. –  Andy W Mar 8 '12 at 21:14
    
(I meant to question, not chastise, my apologies). Copyright law is very vexed and varies from country to country. Even deciding if something is data or not is a matter of interpretation. For example, Canadian law specifically includes maps in the list of works that can be copyrighted. What you are wanting to scrape above looks like a map (on its face) yet is built from data. So, if it were in Canada would it be copyrighted or not? –  Sarge Mar 8 '12 at 22:45
    
US law is a bit different to most other jurisdictions because the case law on data includes quite a bit (specifically, the case was about yellow pages type data). Your question comes across as "help me get the data from this site". –  Sarge Mar 8 '12 at 22:47
show 5 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.