1

I'm working with an xyz file that has 53000000 points, I have only been able to see this in Microsoft Access, because in text it says that the amount of data is too large. So what I've tried to do:

  1. Change .xyz to .txt ---> 3D Analyst tool, conversion, from file, ASCII 3D to Feature Class: So when I try to convert to natural neighbor I can't see the z data so I cannot do the interpolation. I also try using the Tin tool, but it just say that an error happened when it try to draw it.

  2. I've opened .txt with Microsoft Access (it's when I saw the amount of data that I have), and I saved this table in .mdb, but I have no idea on how to use this to make an interpolation.

Please I need some help to work with this data!

  • 1
    Have you tried getting a small portion of the text file to work as you'd like? Once you get that working, you can begin processing the 53M XYZ values in chunks. Most computers can't handle that amount of raw data. – Paul Jul 29 '13 at 15:53
  • How do you suggest me to get small portion of the data? :( i didn´t try that, but i think that you are right about the handling that amount of raw data. – Romina Jul 29 '13 at 16:03
  • How are your points spatially distributed? Are they sequences along the ship tracks? What is the mean distance between points in a track and the distance between tracks? What is your desired cell size? – nadya Jul 30 '13 at 3:12
1

Use Linux. In shell (or bash) you can check that data set very easily (with "less" or even "vim"). There you can install GMT (Generic Mapping Tools) and there you have several tools to examine/grid/clean/etc your points, in fact 53 million points are nothing, I've played with more than 500 million points without problems, you just need a lot of ram to run things faster (more than 8 GB). If you stick with Windows, you can install cygwin and there install GMT, or install GMT in Windows. I, however, suggest always using Linux for such big data sets.

Hope this helps,

0

Since you are running ArcMap, you should have Python installed. You can save this as a .py file and it will copy the first n lines to a new textfile. I'd run it on a small, unrelated test file first so you know how it works.

infile = raw_input("Enter the path to large textfile: ")
outfile = raw_input("Enter the path to new, smaller textfile: ")
numlines = raw_input("Enter how many lines you want to extract: ")

from itertools import islice

with open(infile, "r") as larger:
    with open(outfile, "w") as smaller:
        [smaller.write(x) for x in list(islice(larger,int(numlines)))]

Edit:

I created a txt file of 53M XYZ values, and the above sucessfully copied 1M lines in .8 seconds. Granted, 1M is still pretty large, but plain old notepad was able to open it. I'd recommend Notepad++ though. It handles large files quite well.

0

To split the data file into pieces you can use a small free program Hjsplit. Then you can play with it and find how exactly you wish to process and interpolate it. When you see that the desired interpolation works well for small portions but fails for the entire dataset, then what you need is a powerful computer with more RAM. Try bigger cell size firstly. (I also had a similar problem with a big sonar dataset.)

If you cannot use another computer you have options:

  • To interpolate parts of the area with some overlay and then to mosaic them (for example split your area into 4 or 9 parts). You can input all the points and specify the output extent; or split the points.

  • To decrease the amount of data. If the points are ordered exactly in the way how they were collected (along the ship track, no interruptions), maybe you can average every n points (rows in the file). If the final desired resolution of your interpolation is much lower than data (I mean, if you can get tens of points in every raster cell), you can delete every second or n-th point (some scripting may be needed). Of course deleting is losing the data, but it can be acceptable if the goal is just visualization. (I decreased my dataset prior to interpolation by overlaying a net and computing spatial averages, with Python.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.