2

I have a few large datasets (about 10 million points) which I am buffering, then joining, dowgrading and finally buffering and dissolving. This is all in a model which works fine with sample smaller datasets. Details are in Buffer point dataset based on compared value

Is there a way to split the point dataset and then process it through to the dissolved buffer and then re-join the files? The key question here is to split a "square" dataset into blocks so that the re-join does not create jagged edges. OR do I just have to swith to another software as in Improving upon ArcGIS Buffer/Dissolve performance??

The model builder python code is below. BUT FIRST THE ERROR IS

Executing (Buffer): Buffer P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Longlands\TRC_Longlands.gdb\TRC_Longlands_pts P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Longlands\TRC_Longlands.gdb\TRC_Longlands_buf "30 Meters" FULL ROUND NONE # Start Time: Thu Mar 08 12:54:21 2012 Succeeded at Thu Mar 08 14:42:55 2012 (Elapsed Time: 1 hours 48 minutes 34 seconds) Executing (Spatial Join): SpatialJoin P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Longlands\TRC_Longlands.gdb\TRC_Longlands_pts P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Longlands\TRC_Longlands.gdb\TRC_Longlands_buf P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Longlands\TRC_Longlands.gdb\TRC_Longlands_join JOIN_ONE_TO_ONE KEEP_ALL # WITHIN "30 Meters" # Start Time: Thu Mar 08 14:42:56 2012 ERROR 000426: Out Of Memory Failed to execute (Spatial Join). Failed at Thu Mar 08 15:01:31 2012 (Elapsed Time: 18 minutes 35 seconds)

# ---------------------------------------------------------------------------
# xy_downgradedSS.py
# Created on: 2012-03-08 11:05:07.00000
#   (generated by ArcGIS/ModelBuilder)
# Usage: xy_downgradedSS <Imported_XY> <Buffered_tower_range> <XY_Feature_Class> <INPUT_txt_file> 
# Description: 
# ---------------------------------------------------------------------------

# Import arcpy module
import arcpy

# Load required toolboxes
arcpy.ImportToolbox("Model Functions")
arcpy.ImportToolbox("P:/2011/Job_237_BushfireHazard_Datasets/Working/post_erratum/ProcessingModels.tbx")

# Script arguments
Imported_XY = arcpy.GetParameterAsText(0)
if Imported_XY == '#' or not Imported_XY:
    Imported_XY = "%Path%\\%Name%.gdb\\%Name%_layer" # provide a default value if unspecified

Buffered_tower_range = arcpy.GetParameterAsText(1)
if Buffered_tower_range == '#' or not Buffered_tower_range:
    Buffered_tower_range = "%Path%\\%Name%.gdb\\%Name%_buf" # provide a default value if unspecified

XY_Feature_Class = arcpy.GetParameterAsText(2)
if XY_Feature_Class == '#' or not XY_Feature_Class:
    XY_Feature_Class = "%Path%\\%Name%.gdb\\%Name%_pts" # provide a default value if unspecified

INPUT_txt_file = arcpy.GetParameterAsText(3)
if INPUT_txt_file == '#' or not INPUT_txt_file:
    INPUT_txt_file = "P:\\2012\\Job_044_DM_Radio_Propogation\\Working\\FinalPropogation\\TRC_Longlands\\trc_longlands.txt" # provide a default value if unspecified

# Local variables:
v_Name__join = Buffered_tower_range
v_Signal_Strength__2_ = v_Name__join
Signal_Strength_Added__2_ = v_Signal_Strength__2_
Buffered_tower_range__3_ = Signal_Strength_Added__2_
Source = XY_Feature_Class
Source_Added = Source
v_Signal_Strength = Source_Added
Signal_Strength_Added = v_Signal_Strength
Path = INPUT_txt_file
Name = Path
v_Name__gdb = Name
v_Path_ = "%Path%"

# Process: Parse Path
arcpy.ParsePath_mb(INPUT_txt_file, "PATH")

# Process: ParseName
arcpy.gp.toolbox = "P:/2011/Job_237_BushfireHazard_Datasets/Working/post_erratum/ProcessingModels.tbx";
# Warning: the toolbox P:/2011/Job_237_BushfireHazard_Datasets/Working/post_erratum/ProcessingModels.tbx DOES NOT have an alias. 
# Please assign this toolbox an alias to avoid tool name collisions
# And replace arcpy.gp.ParseName(...) with arcpy.ParseName_ALIAS(...)
arcpy.gp.ParseName(Path)

# Process: Create File GDB
arcpy.CreateFileGDB_management(v_Path_, "%Name%", "CURRENT")

# Process: Make XY Event Layer
arcpy.MakeXYEventLayer_management(INPUT_txt_file, "longitude", "latitude", Imported_XY, "GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119521E-09;0.001;0.001;IsHighPrecision", "")

# Process: Copy Features
arcpy.CopyFeatures_management(Imported_XY, XY_Feature_Class, "", "0", "0", "0")

# Process: Add Field (4)
arcpy.AddField_management(XY_Feature_Class, "Source_Ch", "TEXT", "3", "2", "50", "", "NULLABLE", "NON_REQUIRED", "")

# Process: Calculate Field (4)
arcpy.CalculateField_management(Source, "Source_Ch", Name, "PYTHON", "")

# Process: Add Field (2)
arcpy.AddField_management(Source_Added, "Signal_Strength", "FLOAT", "3", "2", "10", "", "NULLABLE", "NON_REQUIRED", "")

# Process: Calculate Field (3)
arcpy.CalculateField_management(v_Signal_Strength, "Signal_Strength", "re_score(!Rx!,f2)", "PYTHON_9.3", "f2='99'\\ndef re_score(my_field,f2):\\n\\n if (my_field*1) < 14:\\n  f2 = '0'\\n elif (my_field*1) < 24:\\n  f2 = '1'\\n elif (my_field*1) >= 24:\\n  f2 = \"2\"\\n \\n return f2")

# Process: Buffer
arcpy.Buffer_analysis(Signal_Strength_Added, Buffered_tower_range, "30 Meters", "FULL", "ROUND", "NONE", "")

# Process: Spatial Join
arcpy.SpatialJoin_analysis(v_Signal_Strength, Buffered_tower_range, v_Name__join, "JOIN_ONE_TO_ONE", "KEEP_ALL", "", "WITHIN", "30 Meters", "")

# Process: Add Field (3)
arcpy.AddField_management(v_Name__join, "Downgraded_SS", "FLOAT", "3", "2", "10", "", "NULLABLE", "NON_REQUIRED", "")

# Process: Calculate Field (2)
arcpy.CalculateField_management(v_Name__join, "Downgraded_SS", "re_score(!Signal_Strength!, !Signal_Strength_1!)", "PYTHON", "def re_score(f1,f2):\\n    if f1 < f2:\\n        out1 = f1\\n    else:\\n        out1 = f2\\n    return out1")

# Process: Buffer (3)
arcpy.Buffer_analysis(Signal_Strength_Added__2_, Buffered_tower_range__3_, "30 Meters", "FULL", "ROUND", "LIST", "Downgraded_SS")
Improve this question
12
  • Your title mentions Issues but not what they are - are you getting an error? If not, then while such processing would inevitably have produced errors up until about 9.1 or 9.2 I would think this a chance to just work, even on large data, at 10.0. If not, then tiling your vector data could be commenced by using the Split tool if you have an ArcInfo license (wish it was also available in ArcView:).
    – PolyGeo ♦
    Commented Mar 8, 2012 at 1:27
  • sorry - Out of memory error or just crashes arcgis. Also I have found with other processes that splitting and rejoining actually speeds up the process, so I was wondering if there were ways to split the dateset.
    – GeorgeC
    Commented Mar 8, 2012 at 1:30
  • No prob - have you got access to Split from ArcInfo license? If not you may need to run a cursor through an index layer and select out each polygon to use with Intersect to accomplish the same result with ArcView.
    – PolyGeo ♦
    Commented Mar 8, 2012 at 1:35
  • thanks. The issue with Split is setting the split zone as each radio tower range that we are mapping is different, so will have to create an extent (maybe feature envelope to polygon or in python) and then split that into maybe 4. So it's about 120kmx120km square and each block will then be 40km x 40km. I guess data driven pages->grid index features should work.
    – GeorgeC
    Commented Mar 8, 2012 at 1:58
  • Another useful ArcInfo tool to consider when creating the index if you need tiles of different sizes is Dice (which was new at 10.0 help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//…)
    – PolyGeo ♦
    Commented Mar 8, 2012 at 2:00

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.