0

I have a large shapefile with 92 million points which I had converted from the pixels of a Brazil land cover map. I want to add latitude and longitude fields to the shapefile, and I did so using the Add XY Coordinates tool, but the process not only fails but also messes up my other fields. Below is a screenshot of what I get after running the process:

enter image description here

So POINT_X and POINT_Y are supposed to be the longitude and latitude added by the Add XY Coordinates tool. The landcover field was originally 50 for all the points seen here. The fields named "longitude" and "latitude" were mistakes but they were originally all 0. As seen here, the Add XY Coordinates tool not only failed to add the coordinates correctly (these values don't make sense), but they also changed the values of other fields - every landcover value became 0 and every latitude value turned from 0 to these weird numbers.

Why is this happening and what I can do to prevent this?

I had waited more than 3 hours for this process to run.

I also tried another method, i.e. the Calculate Geometry function in the attribute table, but it messes up the fields in a similar way (though the values it produces are different).

Improve this question
5
  • 1
    Do you really have such large shapefile? The maximum is estimated to be about 70 million points gis.stackexchange.com/questions/286316/….
    – user30184
    Jun 11, 2021 at 14:06
  • @user30184 The answer you linked recommended against using shapefiles - do you know what are some better alternatives that can fulfill the same function?
    – Ryan Ma
    Jun 11, 2021 at 14:11
  • 1
    Create a file geodatabase, import your shapefile in it and try calculating again
    – J.R
    Jun 11, 2021 at 14:46
  • 1
    The shapefile is corrupt and not usable. Convert your raster to a file geodatabe or Enterprise geodatabase table instead (with the latter you could access the ST_X() and ST_Y() without badly fragmenting the table by adding those columns.
    – Vince
    Jun 11, 2021 at 15:50
  • 2
    The better alternative would probably be to keep using raster, and instead of asking how to convert a raster into a vector grid of points, to ask about the aspect of the raster workflow that makes you think you need a vector datatype. Raster formats are good at representing continuous fields...
    – alphabetasoup
    Jun 14, 2021 at 1:50

1 Answer 1

Reset to default
2

Shapefile is limited to 231-1 bytes in the .shp and .dbf, which limits you to 76,695,840 2D point features (unless the dBase-III+ file has a record width exceeding 28 bytes, in which case the .dbf is the limiting factor), so 90 million point features have already overtopped the shapefile format specification (making the dataset corrupt and unusable). Adding two wide floats to the dBase table likely mangled it as well.

You'd need to start over with file or Enterprise geodatabase, which have much larger maximums.

Managing tables with tens to hundreds of millions of features is an exercise in frustration. They're always slow, and if not optimally maintained, can be unusably slow (spatial fragmentation can impact the effectiveness of the spatial index). I've worked with tables with 60M-1000+M rows, but it took extra effort to make the 60M-row table performant (usability wasn't a requirement for the big-big table, so I wasn't asked to try).

While file geodatabase can store very-large tables, it doesn't support partitioning the way true RDBMS or NoSQL databases do, which is why I try not to use FGDB for row counts over 20M features.

Raster datasets are far more efficient for processing tens to hundreds of millions of pixels. I recommend careful consideration of the risks before vector conversion.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.