I am consuming a json web-service with NumPy Array to Feature, I have many nested try/except blocks to consume the data/find errors/etc. The script is functional, however I am having trouble seeing where I am obviously not accounting for a loop to consume the data.

Here is a crude example of my code structure; I am writing the first json object that I loop through to my db, although hundreds are returned. How can I adjust my code to account for all of my loops and incoming data?

import json
import jsonpickle
import requests
import arcpy
import numpy as np
import time
import datetime
import logging
import logging.handlers
import smtplib

#overall exception handling
class TlsSMTPHandler(logging.handlers.SMTPHandler):
     def emit(self, record):
        import smtplib
        import string # for tls add this line

            do something

logger = logging.getLogger()

#arcpy e-mail handling
    def sendResultEmail(msgContents, success_TF):

        if arcpy.Exists(fc):
           requests stuff
    except requests.exceptions.ConnectTimeout as e:
            print "Too slow Mojo!"

    if last_run != now_minus_5:
            print "this block is activated"
        items = []
        for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:
            SRAddress = sr['SRAddress']
            # SRType = sr['SRType']
            if sr['Latitude'] != '':
                y = sr['Latitude']
            if sr['Longitude'] != '':
                x = sr['Longitude']
            if sr['Latitude'] == '':
                blanky = sr['Latitude']
            if sr['Longitude'] == '':
                blankx= sr['Longitude']
                print blanky
                print blankx

            #many try except blocks that consume data
                locationEwastePrevious = ''
                for sr in ElectronicWaste:
                        for ewastelocation in ElectronicWaste['La311ElectronicWaste']:
                                            locationewaste = ewastelocation['CollectionLocation']
                                            if locationEwastePrevious != locationewaste:
                                                locationEwastePrevious = locationewaste

                print "No Electronic Waste Types"

                itemEwastePrevious = ''
                for sr in ElectronicWaste:
                        for ewastetype in ElectronicWaste['La311ElectronicWaste']:
                                        itemEwaste =  ewastetype['ElectronicWestType']
                                        if itemEwastePrevious != itemEwaste:
                                            itemEwastePrevious = itemEwaste
                print"No Ewaste items"

                iteminfoewasteprevious = ''
                countEwastePrevious = ''
                for sr in ElectronicWaste:
                        for ewastecount in ElectronicWaste['La311ElectronicWaste']:
                                        countEwaste = ewastecount['ItemCount']
                                        if countEwastePrevious != countEwaste:
                                            countEwastePrevious = countEwaste
                                            iteminfoewaste = '{0},  {1}, '.format(countEwastePrevious, itemEwastePrevious)
                                            if iteminfoewasteprevious != iteminfoewaste:
                                                iteminfoewasteprevious = iteminfoewaste

            ItemDesc = BulkyItemInfo + DACItemInfo
            outputobjects= int(decoded2['Response']['NumOutputObjects'])

            dt = np.dtype([('Address', 'U40'),
                        ('LatitudeShape', '<f8'),
                        ('LongitudeShape', '<f8'),
                        ('Latitude', '<f8'),
                        ('Longitude', '<f8'),
                        ('SRNumber', 'U40'),
                        ('FirstName', 'U40'),
                       ('LastName', 'U40'),
                       ('HomePhone', 'U40'),
                        ('CreatedDate', 'U128'),
                        ('UpdatedDate', 'U128'),
                        ('ItemDesc','U128' ),




            sr = arcpy.SpatialReference(4326)

            arr = np.array(items,dtype=dt)
            NumPyArray = arcpy.da.NumPyArrayToFeatureClass(arr, fc, ['longitudeshape', 'latitudeshape'], sr)

        print "this does not work"
except arcpy.ExecuteError:
            strResult += arcpy.GetMessages()

Output: I am expecting 187 different types for this request. This output writes correctly to my FC.

[ (u'4437 1/2 W LOCKWOOD AVE, 90029', -118.290563, 34.08883, -118.290563, 34.08883, u'44', u'1-7868751', u'', u'', u'', u'', u'2015-03-23 14:35:39', u'2015-05-26 06:39:15', u' 1, Bird Cage (Plastic)  ')]
  • 1
    items appears to be getting appended to only once (it's not within any of your for loops as far as I can see). Could possibly be an indentation issue, as there is a confusing mix of indentation here. – DWynne May 26 '15 at 17:30
  • The indentation is a bit confusing but entirely necessary, how would I assure that items is appeneded to for the count of my returned records? – Geoffrey West May 26 '15 at 17:53
  • I disagree. Any decent IDE will throw up multiple objections to the pattern of indentation in the code you listed above. My hunch is that items.append is supposed to be within a for loop, but it isn't right now. – DWynne May 26 '15 at 17:55
  • I guess more to the point, if you standardized on 4 spaces per level of indentation, it would be easier to understand the flow of the code. – DWynne May 26 '15 at 18:00
  • 1
    Please see the complete code example here; codepad.org/Dp1ZiViV – Geoffrey West May 26 '15 at 18:00

A few things here, not all of them might be contributing to the problem, but all of them strike me as a possible issue.

arcpy.da.SearchCursor gets opened, but only one record is ever accessed (there is no for loop on cursor, or multiple .next() calls.)

with arcpy.da.SearchCursor(aTable, return_flds, where_clause=where_str, sql_clause=sql_clause) as cursor:
    last_row = cursor.next()

Also for each sr from this line:

        for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:

items gets updated and then written out with the NumPyArrayToFeatureClass line. I believe the way the logic is set up, you'll never get more one item in items.

This pattern of logic below also seems off (there are 3 consecutive cases). As updateRow is only called after the entire fc has been iterated over. Maybe updateRow should be within the for loop.

            cursor = arcpy.UpdateCursor(fc)
            for row in cursor:
                row.setValue(CreatedDate, row.getValue("CreatedDate"))

This too for a similar reason, as only one row will get inserted because all the setValue and insertRow calls come outside of the for loop.

            rows = arcpy.InsertCursor(aTable)

            for x in xrange(0, outputobjects):
                row = rows.newRow()
            row.setValue("Success", 'YES')
            row.setValue("Fail", 'NO')
            row.setValue("Time", Start)
| improve this answer | |
  • All of the curosor functions are designed to query the last rows, so that is correct. However, I am not certain that your explanation is correct for items While designing this I tested the consumption of data without all of the expection handling and it works, meaning that all features/JSON objects are written to my FC. I haven't been able to see what the limiting factor is between the two scripts, the indentation does make a bit confusing. Please see this script which functions correctly without the exception handlers. codepad.org/Gzb3px1R line 500 specifically. excuse comments. – Geoffrey West May 26 '15 at 19:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.