The movement breaks the invisibility of "One With Shadows".
The Eldritch summon "One With Shadows" says:
When in an area of dim light or in the dark, you can use your action to become invisible until you move or take an action or a reaction.
The accent is mine.
If you move, whether it's your turn or out of combat, the invisibility ends.