entities – Limited editing of just the order of entity reference field

I would like to give some users the opportunity to edit the ORDER of a field of entity references holding multiple references– but not the references themselves.

One option would be to post process the field and turn off the editability of the form element. But this seems like to create the form element and then disable it.

Is there a better way?