Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I do have DEM approx. 40x5km with cell size 2x2m, which gives approx. 50 000 000 points.

I need to export it to .xyz (1pt = 1 line in format x y z) file in order to use it in MIKE21.

It seemed like there is loads of tools that will do it (I do have access to Arcinfo 9.3.1 and 10), though I'm stuck for 2 days already. I tried some solutions and all have failed.

I did: 1. SA/EXTRACTION/SAMPLE, receiving table, saving it as .dbf, and trying to import it in to Access 2007. I could not import .dbf produced in Arc into Access. Exporting table to .txt gives me always error.

  1. Raster Calculator and Sample(). This worked great. Raster converted in 5 minutes producing 1.2GB file, but in order: ZXY not XYZ. MIKE do not allow to change order of columns, so I need to change order of columns. No clue how to do it. Tried Access 2007 again, but failed on import data.

  2. Script Raster2XYZ ver. 1.1 from ESRI website (in VB). Can not install it properly now (gives an error), I used it before and it worked 6 months ago. However I remember that it worked really slow back then. I am not much of a programmer and can not find the bug.

  3. Grass GIS and r.out.xyz have bug. Problem with double or float writing(can't remember now).

  4. I have R script which does export .asc to .xyz. It take 2.5h to export 40 000 000 cells. Output file is 4.3GB. MIKE will not swallow it.

Any ideas how to export such raster to .xyz? Or how to reorder file from pt.2 to receive XYZ instead ZXY?

I always used Access to convert data (ie. .dbf to .txt(.xyz)), but now it seems like no option. Is there any other solution?

I do have HP workstation Z600 (Xeon 2.4Ghz, 8GB RAM, GF Quadro FX 3800).

Any sugestions?

share|improve this question
    
All answers are helpful. It is a pity that I can choose only one as accepted, thanks again. –  Tomek Jul 25 '11 at 12:05
add comment

6 Answers 6

up vote 2 down vote accepted

You say that

  1. "Raster Calculator and Sample(). This worked great. Raster converted in 5 minutes producing 1.2GB file, but in order: ZXY not XYZ. MIKE do not allow to change order of columns, so I need to change order of columns. No clue how to do it."

To reorder the columns in the text file I'd suggest regular expressions. You can use regular expressions in various text editors (I use EditPad Pro), or in Python, or in the command line tool sed available for Windows as well.

Your regular expression

I assume your "Raster Calculator and Sample()" output looks like this, where 567.89 is the z-value you want to shift to the end of each line:

567.89   123.45   345.67

The regular expression to fetch the line with the three coordinates would look like that:

^([0-9]+\.[0-9]+)\s+([0-9]+\.[0-9]+)\s+([0-9]+\.[0-9]+)$

where

^        means beginning of line
[0-9]+   means one or more digits
\.       means a point (escaped)
\s+      means one ore more whitespace characters
$        means end of line
brackets create backreferences

You replace the line by

\2   \3   \1

where the numbers refer to the three backreferences. In the above example they are separated by blanks. If you want to separate them by tabs you use: 2\t\3\t\1 - whatever MIKE21 likes best.

The result of the replacement would be

123.45   345.67   567.89

If you use sed for Windows

On the command line you may use sed for Windows. Sed is a stream editor and won't have problems with large files.

The sed for Windows syntax for search and replace with regular expressions is

sed -i "s/SEARCH/REPLACE/g" textfile.txt

In sed you have to escape the round brackets and the plus sign. So if your original text file is called "cells.txt", the regular expression wrapped in the sed command line expands to:

sed -i "s/^\([0-9]\+\.[0-9]\+\)\s\+\([0-9]\+\.[0-9]\+\)\s\+\([0-9]\+\.[0-9]\+\)$/\2   \3   \1/g" cells.txt

This expression performs the replacement in place, so make a backup of your original file. Sed will create a temporary file beside your original file, which you can delete afterwards, just make sure there is enough space on your disc.

Additional resources for regular expressions

stackoverflow

Jan Goyvaerts' Regular-Expressions.info

share|improve this answer
    
Great stuff. Sample() in Raster Calculator (available in 9.3.1, could not find it in ver. 10) function do produce smallest .xyz files. Regular expression does the trick really good. Following your links I did some amendments of your expressions for my needs and it worked well. Thanks. –  Tomek Jul 25 '11 at 11:59
add comment

You can use GRASS GIS. I have taken a DEM and exported that much cells into x y z. It takes on my two years old laptop 2min 45sec, creating 1.4GB. The command is:

r.stats -1g elevation > my_xyz.txt
share|improve this answer
    
Works great. r.stats produced a bit smaller file than GDAL. I do not know where the differences come from. Though comparing GRASS GIS and GDAL, GDAL by itself is a bit easier to use. Thanks. –  Tomek Jul 25 '11 at 11:53
add comment

GDAL 1.8 (or later) can convert DEM files to XYZ ASCII file formats.

On a system where GDAL is installed (from source, FWTools, OSGeo4W, etc.), go to the shell and try the following command on your file.dem (or whatever):

gdal_translate -of XYZ file.dem file.xyz

I haven't tested out a raster of this size, but my guess is your xyz file will be close to 2 GB. Also, there are options if you need a header (default is NO), or how to separate columns (default is one space).

share|improve this answer
    
Thanks. Did not know this. This works great, highly recommended for smaller files, as in my case it produced 6GB file (its for 130mln pixels now). –  Tomek Jul 25 '11 at 11:47
add comment

You could try FME. It would do the job.

Simply use a DEM reader (whatever the format is) and write to Point Cloud XYZ format. Throw in a GeometryCoercer transformer in between to coerce the data to a point cloud.

I just tried this on a 1201 x 1201 file (1.4m points) and it took only 24 seconds, so I am hopeful it would handle your 50m points in a not too unreasonable time.

FME Workspace

Unfortunately I don't think the Interoperability Extension in ArcGIS would do this. It doesn't handle raster data. But if it was an Esri source format, you might be able to read it with ModelBuilder and push it into an ETL model that way.

Either way, though, I'm not sure how the output would (or could) be any smaller than the file output by your R script. So you might be stuck on the sheer volume of data.

share|improve this answer
add comment

You could also tile the DEM and process the tiles to XYZ. If you're pushing 50,000,000 lines out to a text file, the output is always going to be big. Tiling may be your only option to keep the text file sizes down to a minimum.

Digesting smaller inputs may decrease your load times too. At least you will be able to track your progress as you load the tiles into MIKE21.

Just a though.

share|improve this answer
    
In deed spliting DEM is the only option here. Thanks. –  Tomek Jul 25 '11 at 11:49
add comment

You can try StarSpan. I recently striped an image file with 49 982 109 pixels into a csv.I only had x and y, but it has a lot of options.

share|improve this answer
    
Thanks, did not use it,but it may be useful in the future. –  Tomek Jul 25 '11 at 11:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.