I am trying to make a custom ToolButton but I am having some problems getting correct measurements.

I have drawn a line (using the normal line tool) and set its length to 1000m. (Double-clicking it and measuring it with the ruler tool both show it to be exactly 1000.00m)

However when I use my tool and drag from one end of the line to the other it gives a distance of 999.8892m. I'm guessing it's some kind of projection issue but I'm using the same settings throughout the system... Where am I going wrong?

Include "mapbasic.def"
Include "icons.def"

Declare Sub Main
Declare Sub LineTool
Declare Sub ExitProg

Sub Main
  Alter ButtonPad "Drawing"
        Icon MI_ICON_ARROW_19
        HelpMsg "Click on map to draw a line"
        DrawMode DM_CUSTOM_LINE
        Calling LineTool

  Alter Menu "Tools" Add
    "Close Line Tool" Calling ExitProg
End Sub

Sub ExitProg
  End Program
End Sub

Sub LineTool
  Set Distance Units "m"
  Set Coordsys Earth

  Dim x, y, x2, y2 as float

  x = CommandInfo(CMD_INFO_X)
  y = CommandInfo(CMD_INFO_Y)
  x2 = CommandInfo(CMD_INFO_X2)
  y2 = CommandInfo(CMD_INFO_Y2)
  print x & ", " & y & ", " & x2 & ", " & y2 

  If x <> x2 and y <> y2 Then
    print Distance(x, y, x2, y2, "cm")
    print "That wasn't a line."
  End if
End Sub
  • If your tool involves interacting with visible points on the screen with a mouse, please remember that the mouse is internally giving coordinates to the nearest pixel. On a high-resolution screen there will be 0.3 to 1.0 meters per pixel. Anything less is just measurement error. When measuring the length of a line segment with a mouse, there will be two independent errors of this amount at the two endpoints, resulting in a magnification of the imprecision by sqrt(2). This leads to 0.5 to 1.4 meters imprecision in the total length. In particular, 1000m = 999.8892m at this level of precision.
    – whuber
    Commented Oct 24, 2011 at 17:07
  • @whuber Does snap effect this?
    – Mr_Chimp
    Commented Oct 26, 2011 at 9:24
  • I don't know, but I believe a good snapping routine would snap to vertex coordinates rather than pixel coordinates on the screen. But length measurements on screens are intended to be crude: they're one-off checks, not high-precision calculations. Therefore software developers would feel free to use quick approximate formulas and they might also use a different projection than you expect. The question comes down to what your "tool" is really intended for.
    – whuber
    Commented Oct 26, 2011 at 14:43

3 Answers 3


MapInfo has two Distance functions. CartesianDistance and SpericalDistance (and Distance which would be the same as SpericalDistance) Try to use CartesianDistance() in stead of Distance()

  • Thanks Peter! I'm still a little confused though. It was returning -1 (meaning it's a long/lat coord system which is fair enough). However switching to cartesian in all the menus I could find made no difference. I got it to work by doing a "set coordsys" statement using the coordsys clause for British national Grid. But this is the projection that all my layers and system should be using anyway. What did "set coordsys" do that the menu system didn't? Thanks again.
    – Mr_Chimp
    Commented Oct 24, 2011 at 13:26
  • 1
    @Mr_Chimp I'm pretty sure that MapBasic apps can have their own coordsys that isn't the same as the systems, which is why you have to tell it what to use. I haven't done MapBasic in a long time so I might be wrong.
    – Nathan W
    Commented Oct 25, 2011 at 1:38
  • 1
    Yes, the coordinate system used by a MapBasic application is independent of that used in a map window. To ensure consistency, you should replace the Set Coordsys Earth line with set coordsys window frontwindow(). This will force MapBasic to use the coordinate system in use in the active window, in this case the map window that the tool was used on. An inappropriate coordinate system will affect whether the cartesian or spherical technique is used by the Distance function, and hence may be the cause of the discrepancy. Commented Oct 25, 2011 at 16:24
  • @Nathan W - That would make a whole bunch of sense. Thanks!
    – Mr_Chimp
    Commented Oct 25, 2011 at 16:25

As Peter says, MapInfo has two different approaches to distance measurements - cartesian, or spherical, the latter taking into account the curvature of the earth. The approach used can be set for individual map windows by altering the settings found under Map > Options, the default setting for new map windows can be set under Options > Settings > Map Window

Check which setting is being used when drawing and measuring the line in the Map window - you should see the words Spherical or Cartesian appearing after the measurement. Check whether this corresponds to the method being used by your MapBasic program by choosing the appropriate function from those described by Peter.


Could it be because of a rounding issue as the result you give from MapBasic is to 4 decimal places? I also notice you are working out the distance in centimetres.

In MapBasic you can round your numbers using the Round function:

Syntax Round( num_expr, round_to )

num_expr is a numeric expression.

round_to is the number to which num_expr should be rounded off.

  • Giving the distance in metres gives a similarly incorrect result. I changed it to cm in order to test it. Using the ruler when measuring in cm gives a length of 100000 cm which is correct. Using my tool in cm gives 99988.92. I could round the answer but that is just avoiding the problem.
    – Mr_Chimp
    Commented Oct 24, 2011 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.