Currently I'm working on a basic 3D model viewer in MATLAB, which can import .OBJ files (modeled in e.g. Blender). The purpose of this viewer is to test different subdivision schemes (like Catmull-Clark, or Doo-Sabin). A subdivision scheme is basically a method to convert a mesh to a smoother (refined) mesh.
In order to do this properly, I would like to implement some functionality to select Vertices and Edges in my viewer.
Vertex selection isn't very hard -- the way this works is that when I click somewhere in the viewer, MATLAB provides me with two points, the begin- and end points of the line perpendicular to the screen (by clicking on a 2D screen that depicts a 3D scene, you actually click on a line, not a single point). Using this method it is relatively simple to find the closest Vertex.
However, I'm not quite sure how to do the same for an Edge. Imagine that the imported geometry is a simple cube, so 8 Vertices and 12 Edges. If I know click on an arbitrary point in the viewer (i.e. on some line perpendicular to the screen), how can I then determine which of the 12 Edges is the closest one?
[First Edit] Just to be clear, I'm not really looking for a piece of (MATLAB) code, but rather for some theoretical explanation (or just a reference to an article/book/website). Thanks!
[Second Edit] By the way, is this the correct place for such a question? Or might the Mathematics or StackOverflow boards be better suited?