38,062 reputation
992165
bio website quantdec.com
location Northeastern US
age 14
visits member for 4 years
seen 5 hours ago

Consultant (environmental and spatial stats a specialty), expert witness, and teacher. I can be reached through (outdated but still valid) links posted on my web site.

Twitter: @WilliamAHuber // ASA-P website: http://amstatphilly.org/


Why waste time learning, when ignorance is instantaneous?

--T(iger) Hobbes.

For any complex problem there is a simple solution. And it's always wrong.

--[Mis?]attributed to H.L. Mencken by Dava Sobel, Longitude.


20h
comment Arcpy Cell Statistics - maximum number of input rasters
+1 For many reasons, sticking thousands of files in one folder is rarely a good idea. A better organization would have the files arranged into a hierarchy of folders, each containing far fewer datasets. That would practically force one to implement this (better) solution in the first place.
1d
comment Better-matching bounding box by using collection
I'm pretty sure you've seen the packing polygons question. The approach--if not the specific method--I described there will work here, provided you make your question a little more precise and quantitative. You need to stipulate how the program should determine the quality of any approximation. I would guess that fewer rectangles are better and that a smaller error (difference in areas) is better, but what is still missing is some indication of how it should trade off between the two objectives. Can you clarify that issue?
1d
comment How to calculate 90% confidence interval in a raster stack
Since these seem to be a time series of rasters, you ought to consider assessing the possibility of a significant serial correlation coefficient first, because that would (strongly) affect the confidence intervals.
1d
comment Raster diff: how to check if images have identical values?
@Remco The algorithm underlying numpy.unique is going to be more computationally expensive (both in terms of time and space) than most other ways to check that the difference is a constant. When confronted with a difference between two very large floating point rasters that exhibit many differences (such as comparing an original to a lossy compressed version) it would likely bog down forever or fail completely.
1d
comment Raster diff: how to check if images have identical values?
I believe ArcGIS has a built-in capability to build a VAT. There does not seem to be any indication in your answer that you are contemplating a purely Python solution, so you might want to clarify that point.
1d
comment What is an efficient Algorithm to draw a straight line from points?
@Devdatta Sorry for being so terse: I was referring to a lexicographic sort on the x and y coordinates. (Such sorting is often a first step in many algorithms of computational geometry.) It is accomplished simply by sorting on one of the coordinates and then using the second coordinate to resolve any ties. This sort is built in to many programming environments. Finding the endpoints needs only O(n*log(n)) time and O(n) space (or O(n) time with a clever algorithm). It works well here because it automatically handles points that are lined up perfectly vertically or perfectly horizontally.
2d
comment Raster diff: how to check if images have identical values?
It seems likely RAM would not be an issue provided you stick with ArcGIS native operations. It's pretty good with RAM usage when processing grids: internally it can do the processing row-by-row, by groups of rows, and by rectangular windows. Local operations like subtracting one grid from another can operate essentially at the speed of input and output, requiring only one (relatively tiny) buffer for each input dataset. Constructing an attribute table requires an additional hash table--which would be minuscule when only one or two values show up, but could be enormous for arbitrary grids.
2d
comment Raster diff: how to check if images have identical values?
Subtraction seems like a good way to conduct a comparison. However, I believe the histogram would not be very useful in detecting problems with NoData values. Suppose, for instance, that the compression procedure eliminated a one-pixel border around the grid (this can happen!) but otherwise was accurate: all differences would still be zero. Also, did you notice that the OP needs to do this with 7000 raster data sets? I'm not sure he would relish examining 7000 plots.
2d
comment Raster diff: how to check if images have identical values?
One huge way to fool these statistics would be to permute the cell contents (which can happen, and does, when image dimensions are not quite right). On very large rasters neither the SD nor the mean would reliably detect a few small changes scattered about (especially if a few pixels were just dropped). Conceivably they would not detect a wholesale resampling of the grid, either, provided cubic convolution were used (which is intended to preserve the mean and SD). It would seem prudent instead to compare the SD of the difference of the grids to zero.
2d
comment Raster diff: how to check if images have identical values?
Would this work with 32-bit floats? Would building and comparing two tables actually be any faster (or easier) than examining the values of the difference of the two rasters (which in principle should be just zero and NoData)?
2d
comment Is the Azimuth on the equator equal to the one not on the equator?
@radouxju An azimuth can be uniquely defined at all points on any surface of revolution except its poles, if any: that includes the sphere and all ellipsoids. It is given by the oriented angle between a direction and the direction of the meridian at that point. I believe this question is asking for the bearings ("azimuths") of geodesics on ellipsoids. More about this appears after the edit to my answer at gis.stackexchange.com/a/6824. As far as I can tell, that answer fully addresses this question; I do not know why this question was reopened.
2d
comment A global, grid-like projection for creating heatmaps
I'm pushing you a little bit because at bottom this really is a question about projections: grids--at least regular ones--can only exist within a projection onto a surface that is flat (or of negative curvature, but those aren't used in GIS!). Thus, any information you can supply about the intended use of your maps can inform the better answers and help you select among them.
2d
comment How to increase raster extent by filling extent with no data using ArcGIS Desktop without Spatial Analyst?
For display and analysis, ArcGIS will automatically treat all regions outside the raster's extent as NoData, so there would seem to be no reason to do this. Is there a particular process in ArcGIS where physically storing the extra NoData pixels actually makes a difference?
2d
comment Raster diff: how to check if images have identical values?
That looks sweet and simple. I am curious about two details (which, technical though they are, could be crucial). First, does this solution handle NoData values correctly? Second, how does its speed compare with using built-in functions intended for grid comparisons, such as zonal summaries?
2d
comment Raster diff: how to check if images have identical values?
One way to implement raster_diff(old_img, new_img) == "Identical" would be to check that the zonal max of the absolute value of the difference equals 0, where the zone is taken over the entire grid extent. Is this the sort of solution you're looking for? (If so, it would need to be refined to check that any NoData values are consistent, too.)
2d
comment Is there a way to get accurate daily U.S. populations?
Due to the uncertainty in extrapolation, the US Census normally won't even provide annual county population estimates (except for a few extremely large counties).
2d
comment A global, grid-like projection for creating heatmaps
I am sure you respect your users and would not want to present maps that could deceive them or otherwise be misread. When they look at your maps, they will be performing the analysis. What will they be concluding from what they see? If they will be reasoning about densities of variables, you owe it to them to use a projection that is approximately equal-area. If they will be using it to get compass bearings, you need a cylindrical projection; etc., etc. In any event, the MAUP will by highly relevant to anyone who wishes to rely on the data you present.
2d
comment Is the Azimuth on the equator equal to the one not on the equator?
Your readers cannot tell you about your JS function because you do not supply it. For the reason why azimuths are not 90 degrees between points at the same nonzero latitudes, please see gis.stackexchange.com/questions/6822.
2d
comment What is an efficient Algorithm to draw a straight line from points?
@Andre A lexicographic sort would do the same but be more robust and reliable. Once the endpoints (X0,Y0) and (X1,Y1) are found, all points (X,Y) can then be sorted by their dot products with the direction vector; that is, by the values (X-X0)*(X1-X0) + (Y-Y0)*(Y1-Y0).
Aug
26
comment Determine PostGIS geography polygon orientation
A reliable algorithm is not possible, because polylines on the sphere merely divide it into two pieces; neither one of them is distinguished (the way the infinite component in the plane would be). Let's hope an accessible orientation property is stored with the polygon.