Overview

In order to figure out whether your cursor is hitting any of the 3D objects in your scene, you need to do the following:

• Transform to the perspective view used by the vertex shader;
• Loop through all the triangles of all the objects in the scene;
• Make sure any triangle is not degenerate (ie: doesn't have zero area);
• Check to see whether the cursor position is inside that triangle.

The trick to making all the following steps easier is to transform the entire scene using the same perspective transformation that is used by the vertex shader. For any given vertex, first we transform by the object's global transformation, and then we apply the perspective transform, remembering to then divide through by the homogeneous coordinate w:

px py pz pw

←

P

M

x y z 1

Position is then given by: [ p_x/p_w, p_y/p_w, p_z/p_w ].

We loop through the triangles of a mesh that contains n vertices simply by looping through every set of three consecutive vertices:

V0, V1, V2
V1, V2, V3
V2, V3, V4
   ...
Vn-3, Vn-2, Vn-1

We don't want to hit test any triangle that has zero area. A simple test to see whether a triangle has zero area is just to do an area calculation, and see whether the result is zero.

Given three vertices A, B and C, the area of a triangle is given by:o

(B_x - A_x) (A_y + B_y) / 2   +   (C_x - B_x) (B_y + C_y) / 2   +   (A_x - C_x) (C_y + A_y) / 2

A point P is inside triangle (A,B,C) if it is on the same side of all three of the triangle's edges A→B, B→C and C→A.

To compute which side of edge A→B that P lies on, we use the difference vector V = P - A.

We also rotate the vector B-A by π/2, to get:

W = [B_y - A_y, A_x - B_x]

Then we just need to check the sign (negative or positive) of the dot product (V●W).

If this sign is the same for all three edges, then point P is inside the triangle.

For your reference, the code we implemented in class is at code9.zip.