1

Long story short, I'm looking to take 1/3 arc-second 1x1 degree contour shapefiles from the USGS National Map Viewer and transform that into a contoured map of a particular state, but am encountering some significant processing problems along the way. Question is in bold near the end of the post.

USGS The National Map View

To download the maps, I selected all 3DEP Elevation Products, then downloaded the 1/3 arc-second 1x1 degree quads that covered at least part of the state I'm interested in (California). This was easy.

I then used ogr2ogr to merge the contour quads into a large grid with the help of a shell script that I wrote, shown below:

#!/bin/bash

ogr2ogrdir=''
directory='/'
output=''

cd $directory || exit

i=1

while IFS= read -r -d '' shpfile
do
    if [ ! -e $directory/$output ]
        then
            echo "[$i] Creating $output from shapefile $shpfile..."
            $ogr2ogrdir -f 'esri shapefile' $directory/$output "$shpfile"
        else
            echo "[$i] Appending shapefile $shpfile to $output..."
            $ogr2ogrdir -f 'esri shapefile' -update -append $directory/$output "$shpfile"
    fi
    ((i++))
done < <(find . -name '*.shp' -print0)

However, naturally ogr2ogr does not support the generation of files larger than 4GB. California was looking likely to be closer to 6GB. To work around this, I picked an arbitrary minute of latitude to divide California into northern and southern sections, and then merged the maps into two files. This produced two shapefiles of approximately 3GB each, looking like the following images. After some processing to remove contour lines outside of the state boundary (Vector > Geoprocessing Tools > Clip using a state boundary polygon as the clip mask), here is what the two files look like from a screenshot from QGIS below.

Northern CA contour output from ogr2ogr and clipped with QGIS Southern CA contour output from ogr2ogr and clipped with QGIS

What I'd now like to do is merge these two separate files into a single state shapefile of contour lines. But, I'm encountering issues with QGIS. I've tried Vector > Data Management Tools > Merge Vector Layers in QGIS, but sadly the output is not what I'm expecting. As you can see in the image below, some (why?) of the southern CA original quads in the southern CA shapefile are missing in the final merged product.

Merged data in QGIS showing missing quadrangles

I'm not even sure why the artifacting is even occurring along the original quadrangle lines - ogr2ogr supposedly merged them, or at least that's my understanding of the situation.

How can I resolve this issue to produce a clean output of contour lines for the state of California in a single shapefile?

Additionally, I'm noticing at the boundary between my northern and southern demarcation point, I'm getting some misalignment between the contour files. Shown in the image below is the contour lines intersecting at the western point of the state at 41 degrees latitude. Could this be causing the error I'm seeing above?

Mis-aligned contour lines

  • could you use some format that is not Shapefile to avoid the size limitation? – Ian Turton Jun 22 '17 at 9:49
  • @iant Oh, the size limitation is in the Shapefile format? That is ridiculous. – ReactingToAngularVues Jun 22 '17 at 9:55
  • back when Shapefiles were designed 32bit was enough for anyone :-) – Ian Turton Jun 22 '17 at 9:56
  • @iant haha, well, I'm a newb to GIS-related matters, but I'm shocked that shapefiles are still in use if such a severe limitation exists! All I'm really looking to do is produce maps for a Garmin GPS. However most of the programs seem to require the input filetype to be a shapefile. I'm not really sure what else I can do in this situation then... – ReactingToAngularVues Jun 22 '17 at 9:58
  • are you willing to use spatial databases? if so -> yay, postgis! If you plan to stick to QGIS to design your maps and need that same (or additional) data for multiple projects, check out PostgreSQL with PostGIS. even if you have no or limited experience with SQL/RDBMS, the simplest installation/setup of PostgreSQL with the PostGIS extension would suit you well I think. QGIS also offers a native interface to work with data from PostGIS. – ThingumaBob Jun 22 '17 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.