OpenStreetMap Style Editor besides Mapbox TileMill and CloudMade

Question

I'm looking for an online OSM style editor that lets me style OSM ways and nodes by tags. I know CloudMade's editor is no longer available, and Mapbox's does this but has zoom level rendering limits (eg, won't render buildings below zoom level 13).

My goal is to get a map of the USA (or world if I can) that shows where every OSM building has been mapped (ways with building=). And a second map that shows where every OSM building + street address is (ways with addr:housenumber=).

Here's an example map I did using a state OSM extract in Kentucky, processed with Osmosis, and QGIS to render:

https://twitter.com/CivicDataAlly/status/644963898463023108

But unfortunately the North America extract is too large for me to work with, and QGIS can't even open the large Osmosis processed file on my machine. I also set up osm2pgsql and PostgreSQL/PostGIS databases, but again the file sizes and processing times are too huge.

Is there an online tool where I can style buildings at a low zoom level?

---

SOLVED

I finally got the image I needed of all the buildings and buildings with street addresses for all of North America, by using the info in the answer below.

I basically setup postgres and used pgAdmin3 to manage it and then osm2pgsql in the command line with the correct options and styles to load the raw .pbf export into the database, then connect to it with QGIS to visualize it.

Once the data was in postgres it was easy and fast to deal with, even displaying it in QGIS on my modest computer.

Buildings

Here are the stats and what I did for the building footprint layer:

osm2pgsql -U mschnuerle -d mschnuerle --create --slim --flat-nodes nabuildings.bin --cache 18000 --number-processes 3  --style buildings.style  north-america-latest.osm.pbf;

Processing: Node(794783k 1292.3k/s) Way(52669k 32.51k/s) Relation(471970 89.52/s)  parse time: 7507s (2 hours)

node cache: stored: 794783721(100.00%), storage efficiency: 66.31% (dense blocks: 695842, sparse nodes: 243029485), hit rate: 100.00%

Osm2pgsql took 88452s overall (24 hours)

Backup DB size: 4GB

with only this in the buildings.style file (everything else deleted):

way         building           text    polygon
way         way_area           real    linear

Addresses

And here is what I did for only the buildings that had street addresses attached to them:

osm2pgsql -U mschnuerle -d mschnuerle --create --slim --drop --flat-nodes naaddress.bin --cache 6000 --number-processes 4 --verbose --exclude-invalid-polygon --prefix address --style address.style  north-america-latest.osm.pbf;

Processing: Node(794783k 1271.7k/s) Way(52669k 31.67k/s) Relation(471970 37.13/s)  parse time: 14998s (4h)

Osm2pgsql took 17455s overall (5h)

with this in the address.style file (everything else deleted):

way    addr:housenumber       text    polygon
way    way_area               real    linear 
way    building               text    delete

And my machine specs are this, for those asking:

Mac Specs

OS X El Capitan 10.11 Beta
iMac 27" Mid 2011
3.1 GHz Intel Core i5 (4 cores)
25 GB 1333 MHz DDR3
1TB SATA HD 

This is the only map online that shows building footprints at this level, as far as I know. If you find any other ones, please let me know here.

I also plan to create all the Mapbox tiles needed using Tippecanoe and post the map online over the next few weeks.

Answer 1

The short answer is, there's no outstanding way.

Options that won't work well

Hadoop

Hadoop and similar tools aren't the solution, as it's entirely possible to do this type of analysis on a reasonably powerful server. You may not have a reasonably powerful server, in which case Hadoop wouldn't be a good option since it needs a cluster.

If you happen to have a Hadoop cluster and are an expert in using it, it's reasonable, but otherwise it's more development time for no gain.

Vector tiles

Vector tiles don't remove any processing steps, they just allow some of the work to be shared by multiple styles. As you've seen with Mapbox Streets' styling, buildings aren't often in low-zoom vector tiles, so you'd have to generate them yourself.

You could stitch together low-zoom vector tiles, but you'd have to use your own rendering toolchain for that, and it would be complex.

Reasonable options

OSM has about 160 million building ways, 35 million addresss nodes, and 21 million ways with addresses. Most of the last are also buildings.

osm2pgsql

osm2pgsql can handle this on reasonable hardware, if you take care to exclude other data. To do this you want a custom .style file which includes only address and building tags. Starting with empty.style, the suggested starting point, we can get

node,way addr:unit text linear node,way addr:housename text linear node,way addr:housenumber text linear node,way addr:street text linear way building text polygon

Everything below here is copied from empty.style, with the "building" and z_order lines removed. The former is above as an actual column, the latter isn't relevant for this use way_area is included, as it is useful

way abandoned:aeroway text phstore way abandoned:amenity text phstore way abandoned:building text phstore way abandoned:landuse text phstore way abandoned:power text phstore way area:highway text phstore node,way aeroway text phstore node,way amenity text phstore way building:part text phstore node,way harbour text phstore node,way historic text phstore node,way landuse text phstore node,way leisure text phstore node,way man_made text phstore node,way military text phstore node,way natural text phstore node,way office text phstore node,way place text phstore node,way power text phstore node,way public_transport text phstore node,way shop text phstore node,way sport text phstore node,way tourism text phstore node,way water text phstore node,way waterway text phstore node,way wetland text phstore way way_area real linear # This is calculated during import

Save it as buildings.style

You can then import the planet, using something based on the suggested osm2pgsql command line for the planet osm2pgsql -c -d buildings --style /path/to/buildings.style --slim --drop -C <cache size> --flat-nodes <flat nodes> /path/to/planet-latest.osm.pbf

where

• <cache size> is 24000 on machines with 32GiB or more RAM or about 75% of memory in MiB on machines with less
• <flat nodes> is a location where a 24GiB file can be saved.

There's a couple of different options used

--drop gets rid of tables used only during the import and during updates, as I'm assuming you'll update by reimporting

--style /path/to/buildings.style specifies to use the style we wrote above

This will take a day or two on a reasonably powered server.

Once the import is done, there's a couple indexes you can add which will help performance

CREATE INDEX planet_osm_polygon_area_18250_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 18250; CREATE INDEX planet_osm_polygon_area_1140_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 1140; CREATE INDEX planet_osm_polygon_area_71_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 71;

When defining the layers in Kosmtik, Tilemill, or another map design studio, include the condition WHERE way_area > 0.05*!pixel_width!::real*!pixel_height!::real on any polygon layers.

Some things to watch for when visualizing

• Rendering lots of little polygons doesn't work well. That, as well as performance, is why there's the area cutoff for layers.
• It's still going to be slow, but pre-rendering the US should be entirely reasonable
• Test on a small area first

https://switch2osm.org/loading-osm-data/ has more information on setting up PostgreSQL and installing osm2pgsql

libosmium

libosmium is a library for working with OSM data, and probably the best option for operating directly on the planet file.

QGIS or ArcGIS

If using this much data in QGIS or ArcGIS, you probably want to script it, but you could do a more sophisticated analysis.

osm2pgsql multi-backend

This is very similar to the above, but features a more flexable backend, which can result in tables better suited for exporting to another format.

Answer 2

qa.poole.ch renders OSM buildings with and without addresses, nodes with addresses and residential areas without addresses for the world since nearly ages on not very high-end HW. The tiles can be freely used. There is nothing particularly refined about it at all, naturally rendering the low zoom tiles is slow, but not they aren't really interesting enough to expend any effort on them.

Answer 3

As you discovered, there is a price to pay trying to process and render millions, or even tens to hundreds, of millions of objects at low zoom. You will hit limits at some point, or may need to sit out excruciatingly long processing or drawing times.

Unfortunately, many less experienced GIS users have a difficulty grasping this, or even refuse to accept this.

Your question may therefor be unrealistic, unless you live in the luxury position of having "scientific research style" Big Data distributed processing facilities at hand, like clusters managed by Hadoop and the associated eco-system of applications like (No SQL) distributed databases, that are actually designed to handle this type of throughput and do analysis on such vast (tens to hundreds of millions, or even billions) of objects in a timely manner.

An average "household" laptop or desktop, or even 12 core professional workstation, is no equal...

It is not without reasons OSM adopted the concept of landuses like "residential", "industrial" etc, to display build-up areas at low zoom.

There are actually tools developed to put data from a GIS in a Big Data processing cluster. E.g. ESRI developed a number of Open Source ones available here: https://github.com/Esri/gis-tools-for-hadoop

An interesting blog by one of it's developers (Mansour Raad) is here: http://thunderheadxpler.blogspot.com/

Answer 4

I'll disagree with some of the other replies: rolling your own vector tiles and rendering them isn't particularly difficult. You will have to drop some data at low zoomlevels in order to make rendering work properly, but there are tools that handle this simplification for you automatically.

In particular, tippecanoe is a great open source project that has been used to create some of the web's most information-dense maps. You can choose your own min and max zoomlevels and simplifications. Here's an example script I used to create vector tiles for the map on openaddresses.io. There's a bunch there, but it includes the steps for building tippecanoe and normalizing the OpenAddresses dataset into GeoJSON. The actual tippecanoe invocation is a single line.

And hey, for good measure, here's a GL map rendered using that VT source.