difference in efficiency between java int and byte

There is really a difference between an int data type and a byte type, I know it's easier to "understand" in terms of reading and writing code, but the Byte theoretically occupies less storage space, I say "in theory" because somebody once said that the Java virtual machine stored the information in spaces of 4 bytes and that the management of a byte was then identical to the management of an integer, since the remaining 3 bytes were wasted. and in terms of running time, is there really a difference?
As for the code structure that is such a good practice to do in a program, I know that doing it to save 6 or 9 bytes does not make sense, and in that case the best thing to do is to use an int since the backup is really insignificant and it is worthwhile to "sacrifice the readability of the code", but if you manage about 20,000 data, is there a noticeable difference or is it better to continue using int for convenience, which would constitute a considerable amount of data, so that it is really worth using byte and not int?