rotation – Vectors do not define a plane when constructing a RotationMatrix

Edit

Another way is use perturbation.

u = {0, 1, 0};
v = {0, -1, 0};
m = Limit(RotationMatrix({u, v + t*RandomInteger(20, 3)}), t -> 0)

{{0, 0, -1}, {0, -1, 0}, {-1, 0, 0}}

{{-(87/425), 0, -(416/425)}, {0, -1, 0}, {-(416/425), 0, 87/425}}

etc.

Original

We can find an orthogonal matrix satisfied $m^{T}bullet m=mathrm{Id},mathrm{Det}(m)=1$ which transform $(0,1,0)$ to $(0,-1,0)$

m = Array(a, {3, 3});
sol = Solve({Transpose(m) . m == IdentityMatrix(3), 
     Det(m) == 1, {0, 1, 0} . m == {0, -1, 0}})((1));
m /. sol
({0, 1, 0} . m /. sol) == {0, -1, 0}

{{a(1, 1), 0, -Sqrt(1 - a(1, 1)^2)}, {0, -1, 0}, {-Sqrt(1 - a(1, 1)^2), 0, -a(1, 1)}}

True

For example we can set a(1,1)->1/2

{{1/2, 0, -(Sqrt(3)/2)}, {0, -1, 0}, {-(Sqrt(3)/2), 0, -(1/2)}}