1

I've created a new field as a whole integer with a length of 10. I am attempting to use the field calculator to assign values to the newly created field. However, I am certain the numbers aren't being calculated correctly and I'm not sure why. At this point I'm just attempting to do 36*1000000000 and the result QGIS gives is 1640261632.

enter image description here

  • 3
    The output of the calculation 36*1000000000 has 11 digits, so it can't fit in a field that's only 10 digits long. I'm not sure why QGIS is giving you a value of 1640251632, though. – csk Jun 28 '18 at 20:26
  • Also, please edit your question to include what version of QGIS you're using. – csk Jun 28 '18 at 20:27
  • 1
    Another issue I noticed: you mention that you're trying to create a new field, but the field calculator is set to update an existing field. – csk Jun 28 '18 at 20:29
  • I'm using QGIS version 2.18.15. Also, I created a new field and I am using the field calculator to update that field, which is currently blank. Thanks! – Niki Jul 5 '18 at 15:13
1

I did a bit of testing, and the QGIS doesn't seem to allow you to calculate numbers that large. At first I thought this was a limitation of the shapefile format. Then I tested a scratch layer and a geopackage layer, and the limitations were the same. So this seems to be a limitation of QGIS.

Test 1

Entered various calculations into the field calculator, and compared the output preview.

Result: The largest correct calculation it would do is 2*1069999999 (output: 2139999998). Any calculation larger than that gave a wildly incorrect number, sometimes positive, sometimes negative.

A few examples:

  • 3*1069999999 = -1084967299
  • 3*1000000000 = -1294967296
  • 700*1000000000 = -79669248
  • 50*100000000 = 705032704

Test 2

Create an integer field, length 11. Type an 11 digit number into the field in the attribute table. Hit enter.

Result: the number disappears and is replaced with NULL.

Test 3

Label a feature using the calculation 36*1000000000.

Result: label reads 1640261632

enter image description here

Test 4

Do a calculation that internally creates a number larger than 2 million, but the final output should be less than 2 million.

Result: final output is incorrect.

enter image description here

Workaround

A workaround for your problem would be to limit your calculations to output numbers smaller than two million. For example, if you were calculating polygon area in square meters, try doing your calculations in hectares or square kilometers instead. Use a spreadsheet to convert the area into square meters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.