1

My code is:

with arcpy.da.SearchCursor("txtFilePath", ["Field1", "lotsOFOthers"]) as cursor:
  for row in cursor:
    val = "%s" % row[0]
    ... non related code which selects a few values
    newTextFile.write(val + "\n")

My issue is that field 1 is a mix of integer and string values, and sometimes the values that are strings get written out as nulls. There is literally nothing written out for that value. I strongly suspect that this is due to Arc seeing only integer values in the first few records and surmising that the field is type int, and then when it tries to read a string it fails.

My question is, is there a way to make sure this won't fail? I know from experimentation that this works when some or all of the first ten records contain strings; in that case both integer values and string values are written out correctly.

I understand I could read and write csv files without using cursors, but is there a way to do this while still using da.SearchCursor?

  • This is a common problem with CSV files, the field type needs to be set so, from the first few lines ArcGIS assumes the data type based on the values.. If you know the format of your table create a new DBF or geodatabase table with the correct field types and import your CSV into the correct field table with Append resources.arcgis.com/en/help/main/10.2/index.html#//… then cursor through the new table. Alternately you could read the text file as a python file object and for Line in TextFile: values = Line.split(',') for a list of strings. – Michael Stimson Apr 4 '18 at 21:38
  • 4
    Data Access cursors were designed with tables in mind. While you can kludge CSV into something resembling a table, the Python language has a real csvfile handler, which will save you pain in the long run. – Vince Apr 4 '18 at 21:54
  • 2
    As @Vince mentioned, the effort invested in built-in csv module (or other relevant python modules) would be far more well-spent than seeking a resolution by arcpy. – fatih_dur Apr 5 '18 at 4:14
4

Yes, it definitely sounds from your description like it is an issue with ESRI seeing the first several rows as a specific field type and therefore assuming the entire field is that field type. If I remember right, from prior research, I think it scans the first 16 rows... but I could be wrong on that one. Anyway, there is a way to force ESRI to see a CSV field as a specific field type, and it's through the use of a schema.ini file. Information on using schema.ini files is available at http://desktop.arcgis.com/en/arcmap/10.5/manage-data/tables/adding-an-ascii-or-text-file-table.htm in the "Overriding how text files are formatted" section at the bottom of that page. Essentially, you create a file called schema.ini in the same folder as the csv file and it would contain something like

[CsvFileName.CSV]
Col1=Field1 Text

And if you have control over the creation of the CSV file, the other simpler option, if you want a CSV field's values to be read in as text instead of a number, would be to simply enclose all of those values in double quotes. This is pretty standard for CSV files as a way to designate a value as a string, especially if the string may include a (,). So the CSV file may look something like:

Field1,Field2,Field3
"1","Hello",15
"A","World",99
...

Finally, if you are not stuck with using an ArcPy to do this, Python does have a csv module specifically for working with CSV files that may be of value to you. If I remember right it reads each row in as a python list, so you could work with the values pretty similarly to how you would work with a row in a da.SearchCursor. More info on that module at https://docs.python.org/2/library/csv.html

  • That schema thing is totally new to me, and I will try that. Is that the same reason that a schema file is created when I export a table from ArcMap? – Craig T Apr 4 '18 at 22:30
  • I believe you are correct, but am not for sure on that. – John Apr 5 '18 at 14:20
0

Yes, I would echo what was posted about the CSV module in python. It's standard library so you should be able to use it within an arcpy script if you 'import csv'. Then you can iterate through the csv etc and get the values you want. Or alternatively, just use that to read the csv in and write it again to a new file. That may correct errors.

  • I know of the csv module and how to use it, I'm specifically asking if this can be done without that module. – Craig T Apr 5 '18 at 15:54
  • @CraigT, is there a practical reason for your application that you can't use the csv built-in? – Tom Apr 5 '18 at 17:11
  • Two reasons: There are actually a couple hundred lines of code in this cursor and I don't want to rewrite; the rest of it has worked fine for quite a while. Also I'm simply less familiar with the csv module and don't have time right now to learn; for example I don't even know how to select only fields I want to use from the table I'm looking at. – Craig T Apr 5 '18 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.