0

I've been experimenting with scraping spatial data from a web page (www.majidata.co.ke). I've managed to find the source of the vector data from within a webpage though and have saved a chunk as HTML for example this relates to some polygons:39.655756,-4.034769 39.655757,-4.034769 39.655758,-4.034769 39.655800,-4.034788 39.655885,-4.034809 39.655977,-4.034861 39.656021,-4.034878 39.656033,-4.034858 39.656127,-4.034883 39.656194,-4.034850 39.656165,-4.034810 39.656144,-4.034763 39.656085,-4.034735 39.656065,-4.034732 39.656062,-4.034593 39.656045,-4.034521 39.656044,-4.034493 39.656008,-4.034416 39.655976,-4.034420 39.655952,-4.034409 39.655938,-4.034412 39.655929,-4.034398 39.655867,-4.034358 39.655809,-4.034354 39.655798,-4.034312 39.655803,-4.034246 39.655768,-4.034227 39.655681,-4.034251 39.655573,-4.034372 39.655569,-4.034373 39.655534,-4.034409 39.655499,-4.034411 39.655459,-4.034412 39.655409,-4.034399 39.655401,-4.034397 39.655331,-4.034413 39.655247,-4.034424 39.655185,-4.034420 39.655161,-4.034431 39.655092,-4.034426 39.655007,-4.034467 39.654996,-4.034477 39.654991,-4.034478 39.654947,-4.034459 39.654946,-4.034459 39.654931,-4.034462 39.654878,-4.034474 39.654876,-4.034474 39.654816,-4.034461 39.654816,-4.034459 39.654814,-4.034458 39.654738,-4.034457 39.654688,-4.034450 39.654588,-4.034443 39.654584,-4.034452 39.654551,-4.034594 39.654542,-4.034636 39.654540,-4.034636 39.654530,-4.034627 39.654479,-4.034630 39.654378,-4.034657 39.654352,-4.034658 39.654302,-4.034675 39.654302,-4.034676 39.654302,-4.034677 39.654305,-4.034681 39.654304,-4.034780 39.654307,-4.034784 39.654308,-4.034786 39.654281,-4.034901 39.654287,-4.034905 39.654343,-4.034898 39.654366,-4.034885 39.654367,-4.034848 39.654382,-4.034842 39.654379,-4.034844 39.654450,-4.034841 39.654471,-4.034848 39.654503,-4.034874 39.654560,-4.034850 39.654652,-4.034810 39.654709,-4.034767 39.654739,-4.034732 39.654789,-4.034723 39.654800,-4.034713 39.654806,-4.034710 39.654840,-4.034724 39.654907,-4.034722 39.655025,-4.034717 39.655055,-4.034623 39.655084,-4.034614 39.655093,-4.034616 39.655170,-4.034661 39.655182,-4.034663 39.655229,-4.034660 39.655263,-4.034655 39.655338,-4.034669 39.655379,-4.034663 39.655399,-4.034686 39.655510,-4.034747 39.655596,-4.034750 39.655756,-4.034769|BANDARINI|#FF0000!39.660949,-4.031741 39.660951,-4.031718 39.660953,-4.031715 39.661011,-4.031725 39.661032,-4.031725 39.661032,-4.031725 39.661033,-4.031714 39.661062,-4.031671 39.661076,-4.031659 39.661067,-4.031645 39.661035,-4.031595 39.661046,-4.031431 39.661077,-4.031377 39.661074,-4.031320 39.661060,-4.031240 39.661077,-4.031188 39.661064,-4.031057 39.661069,-4.031024 39.661037,-4.030952 39.661034,-4.030820 39.661033,-4.030818 39.660999,-4.030805 39.660981,-4.030804 39.660979,-4.030803 39.660961,-4.030786 39.660960,-4.030786 39.660917,-4.030802 39.660909,-4.030805 39.660903,-4.030805 39.660860,-4.030786 39.660819,-4.030768 39.660816,-4.030768 39.660739,-4.030793 39.660725,-4.030847 39.660735,-4.030916 39.660727,-4.030948 39.660741,-4.030969 39.660738,-4.031017 39.660724,-4.031156 39.660722,-4.031236 39.660732,-4.031300 39.660665,-4.031377 39.660584,-4.031489 39.660575,-4.031641 39.660583,-4.031675 39.660567,-4.031725 39.660587,-4.031841 39.660575,-4.031919 39.660575,-4.031921 39.660577,-4.031940 39.660560,-4.031996 39.660559,-4.032016 39.660537,-4.032058 39.660525,-4.032150 39.660503,-4.032191 39.660558,-4.032219 39.660561,-4.032248 39.660564,-4.032350 39.660568,-4.032391 39.660553,-4.032414 39.660587,-4.032424 39.660646,-4.032415 39.660690,-4.032406 39.660759,-4.032373 39.660754,-4.032280 39.660769,-4.032283 39.660780,-4.032194 39.660793,-4.032155 39.660838,-4.032112 39.660830,-4.031981 39.660827,-4.031967 39.660829,-4.031945 39.660871,-4.031795 39.660867,-4.031770 39.660869,-4.031767 39.660949,-4.031741|PARADISE A|#FF0000!39.660122,-4.032298 39.660153,-4.032287 39.660191,-4.032290 39.660305,-4.032335 39.660370,-4.032356 39.660442,-4.032406 39.660507,-4.032439 39.660529,-4.032435 39.660568,-4.032391 39.660565,-4.032389 39.660562,-4.032349 39.660565,-4.032315 39.660522,-4.032223 39.660511,-4.032217 39.660477,-4.032185 39.660511,-4.032109 39.660500,-4.032088 39.660519,-4.032066 39.660526,-4.032046 39.660552,-4.031983 39.660593,-4.031891 39.660597,-4.031883 39.660586,-4.031782 39.660570,-4.031650 39.660580,-4.031602 39.660551,-4.031509 39.660576,-4.031450 39.660647,-4.031354 39.660736,-4.031297 39.660738,-4.031288 39.660721,-4.031234 39.660719,-4.031175 39.660729,-4.031136 39.660736,-4.031056 39.660743,-4.030945 39.660725,-4.030880 39.660725,-4.030869 39.660716,-4.030805 39.660649,-4.030758 39.660649,-4.030754 39.660595,-4.030750 39.660533,-4.030760 39.660432,-4.030755 39.660328,-4.030770 39.660232,-4.030784 39.660131,-4.030813 39.660016,-4.030825 39.660017,-4.030826 39.659975,-4.030841 39.659883,-4.030839 39.659841,-4.030812 39.659744,-4.030835 39.659701,-4.030856 39.659636,-4.030970 39.659640,-4.031012 39.659597,-4.031058 39.659498,-4.031098 39.659498,-4.031096 39.659468,-4.031105 39.659417,-4.031145 39.659416,-4.031146 39.659416,-4.031146 39.659416,-4.031146 39.659416,-4.031147 39.659383,-4.031142 39.659368,-4.031141 39.659286,-4.031169 39.659197,-4.031164 39.659201,-4.031165 39.659040,-4.031211 39.659032,-4.031221 39.659024,-4.031259 39.658964,-4.031348 39.658954,-4.031433 39.658900,-4.031454 39.658887,-4.031467 39.658816,-4.031530 39.658791,-4.031575 39.658706,-4.031595 39.658675,-4.031583 39.658654,-4.031589 39.658538,-4.031658 39.658486,-4.031699 39.658470,-4.031732 39.658476,-4.031746 39.658504,-4.031809 39.658504,-4.031859 39.658543,-4.031909 39.658582,-4.031956 39.658634,-4.031991 39.658704,-4.032052 39.658707,-4.032059 39.658713,-4.032121 39.658711,-4.032166 39.658735,-4.032222 39.658745,-4.032265 39.658780,-4.032300 39.658778,-4.032305 39.658793,-4.032317 39.658817,-4.032313 39.658898,-4.032307 39.659022,-4.032300 39.659103,-4.032315 39.659228,-4.032285 39.659344,-4.032301 39.659359,-4.032301 39.659517,-4.032306 39.659645,-4.032301 39.659738,-4.032299 39.659786,-4.032314 39.659883,-4.032311 39.660023,-4.032302 39.660122,-4.032298|PARADISE B|#FF0000!

I'd like to be able to parse this to a shapefile (to allow me to plot in QGIS). I can see that there is a string of coordinates, followed by "|name of polygon|" then followed by a hex colour for the polygon outline. I'm assuming it would be possible to use something like the Python library Beautifulsoup to parse this but my Python skills are a little sketchy.

Has anyone had any experience doing anything similar, or know of an easier way to go about this?

5

This only a partial answer and lacks all python but too long to fit into a comment field.

Your data is rather easy to convert into WKT:

  • capture the coordinate part
  • switch commas to spaces and spaces to commas
  • close inside WKT polygon: POLYGON ((...))

    POLYGON ((39.655756 -4.034769,39.655757 -4.034769,39.655758 -4.034769,39.655800 -4.034788,39.655885 -4.034809,39.655977 -4.034861,39.656021 -4.034878,39.656033 -4.034858,39.656127 -4.034883,39.656194 -4.034850,39.656165 -4.034810,39.656144 -4.034763,39.656085 -4.034735,39.656065 -4.034732,39.656062 -4.034593,39.656045 -4.034521,39.656044 -4.034493,39.656008 -4.034416,39.655976 -4.034420,39.655952 -4.034409,39.655938 -4.034412,39.655929 -4.034398,39.655867 -4.034358,39.655809 -4.034354,39.655798 -4.034312,39.655803 -4.034246,39.655768 -4.034227,39.655681 -4.034251,39.655573 -4.034372,39.655569 -4.034373,39.655534 -4.034409,39.655499 -4.034411,39.655459 -4.034412,39.655409 -4.034399,39.655401 -4.034397,39.655331 -4.034413,39.655247 -4.034424,39.655185 -4.034420,39.655161 -4.034431,39.655092 -4.034426,39.655007 -4.034467,39.654996 -4.034477,39.654991 -4.034478,39.654947 -4.034459,39.654946 -4.034459,39.654931 -4.034462,39.654878 -4.034474,39.654876 -4.034474,39.654816 -4.034461,39.654816 -4.034459,39.654814 -4.034458,39.654738 -4.034457,39.654688 -4.034450,39.654588 -4.034443,39.654584 -4.034452,39.654551 -4.034594,39.654542 -4.034636,39.654540 -4.034636,39.654530 -4.034627,39.654479 -4.034630,39.654378 -4.034657,39.654352 -4.034658,39.654302 -4.034675,39.654302 -4.034676,39.654302 -4.034677,39.654305 -4.034681,39.654304 -4.034780,39.654307 -4.034784,39.654308 -4.034786,39.654281 -4.034901,39.654287 -4.034905,39.654343 -4.034898,39.654366 -4.034885,39.654367 -4.034848,39.654382 -4.034842,39.654379 -4.034844,39.654450 -4.034841,39.654471 -4.034848,39.654503 -4.034874,39.654560 -4.034850,39.654652 -4.034810,39.654709 -4.034767,39.654739 -4.034732,39.654789 -4.034723,39.654800 -4.034713,39.654806 -4.034710,39.654840 -4.034724,39.654907 -4.034722,39.655025 -4.034717,39.655055 -4.034623,39.655084 -4.034614,39.655093 -4.034616,39.655170 -4.034661,39.655182 -4.034663,39.655229 -4.034660,39.655263 -4.034655,39.655338 -4.034669,39.655379 -4.034663,39.655399 -4.034686,39.655510 -4.034747,39.655596 -4.034750,39.655756 -4.034769))

The result can be shown with QGIS by dropping the WKT into QuickWKT plugin (experimental).

enter image description here

1
  • hey thanks for the hint. I had played with the QuickWKT plugin and realise I need to parse the data into WKT to use it. I guess I'll have to sit down and figure it out in Python (or R for that matter). Do you know if its possible to scrape straight from the website to a CSV/Shapefile using Python? – marty_c Sep 20 '14 at 1:27
0

I've managed to scrape data and (partially) convert it to spatial data using R.

I can scrape the data using this:

url = "http://www.majidata.go.ke/ajax-maps.php?reg=area&id=30202020116"
test <- readLines(url, warn="F")

Which gives me a character vector containing the coordinates, the name, and a hex colour.

I then split the vector into a dataframe with three columns using the colsplit function from reshape2 and drop the colour column as I don't need it:

test_df <- colsplit(test, "\\|", c("polygon", "name", "colour"))
#first drop the colour column
test_df <- test_df[-3]

Now I put the coordinates in WKT format: and replace commas for spaces as the original poster suggests above:

    #first replace "," with a holding character "~"
test_df$polygon <- str_replace_all(test_df$polygon,",", "~")
        #now replace " " with ","
        test_df$polygon <- str_replace_all(test_df$polygon," ", ",")
        #and finally re-replace the "~" with a space " "; otherwise all spaces would get converted to ","
        test_df$polygon <- str_replace_all(test_df$polygon,"~", " ")
#now add parentheses
test_df$polygon <- paste0("POLYGON((", test_df$polygon, "))" )

Then I create a spatial object from the resulting data.frame using readWKT

I'm a little stuck here now though as I can't get this object out of R. I want to convert this spatial polygon to either a shapefile or better write it to a postGIS table. Not having much luck there so have ask another question here: R PostGIS convert SpatialPolygon to PostGIS table

Thanks for the initial help though:)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.