c # – The most efficient way to get the closest point to a 3d rectangle

Let $ vec {n} $ to be a normalized normalized vector of the plan (for example, the cross product of $ vec {DA} $ and $ vec {DC} $ divided by its length).

Let $ Q = P – ( vec {DP} cdot vec {n}) vec {n} $. (That is to say., $ Q $ is the projection of $ P $ in the plane of $ ABCD $. You already seem to have a function for that. The only missing part is to make sure that $ Q $ belongs to the rectangle.)

Let $ x = ( vec {DQ} cdot vec {DC}) / | DC | ^ 2 $ and $ y = ( vec {DQ} cdot vec {DA}) / | DA | ^ 2 $. (That means we find the place of $ Q $ in a coordinate system in the rectangle that originated in $ D $ and $ (1,0) $ and $ (0,1) $ in $ C $ and $ A $, respectively.)
Yes $ x> 1 $ or $ x <0 $, together $ x = 1 $ or $ x = 0 $, respectively; the same thing for $ y $.

Point $ D + x vec {DC} + y vec {DA} $ is the point you are looking for.

(Of course, this answer still leaves open the question of "the most effective way".)