Let’s assume I’m already storing a BVH acceleration structure per mesh on my engine and I’d like start

making area selection like on many DCC tools out there, for instance, blender.

With blender you can do {rectangular, circle, lasso} selection to select faces and I’d like to understand the maths behind these algorithms. If I was using color picking techniques these sort of algorithms would be obvious but when using BVH+raycasting I’m still failing to understand the algo/maths behind it.

So, could you please explain the logic behind those? For instance, I assume rectangular selection should be easier to implement than circle/lasso, so how would you gather all screen projected triangles living inside the rectangular selection created by the mouse?