I have a directory with a few terabytes of data on server A (unix system, with many symlinks) that I want to back up to my PC (windows 10 system) as my access to server A is going to end soon. In the future, I may need to transfer the backup files to another server (say, server B, unix system) to resume my work there. So here’s the problem:
The best scenario is if I can download the directory on server A as is to my PC, so if I make any changes to the files I have on server A before my access ends, I can sync the changes to my PC directly. The sync can be done easily with softwares like WinSCP, but there’s a problem with the symlinks in the directory. If I simply download the directory on server A to my PC using softwares like WinSCP, which uses SFTP protocol, it will download the symlinks as their target files and I will get many huge (gigabyte-size) duplicate files. To prevent this, I know I can compress the directory on server A into a tar.gz file and backup the the tar.gz file. However, given the huge size of the tar.gz file, it will be very troublesome to access the backup directory on my PC when I need to retrieve any data, since that will involve unziping the entire terabyte-sized tar.gz file on my Windows PC. Also, if I make any changes to the directory on server A after creating the tar.gz, the changes cannot be updated to the tar.gz file easily.
I’m also thinking if opening the bash subsystem on Windows 10 is going to help in this case, since I will then be able (maybe) to somehow “rsync -a” the directory from server A to my PC and access the backup directory using the bash subsystem, so all the symlinks can be preserved that way. However I have zero experience in doing this, so I don’t know both theoretically and practically if this can work.
Hence, may I know if there is a good way of backing up the directory from server A to my PC so that the symlinks are preserved as symlinks and at the same time the backup files can be updated if I make changes to the directory in server A and I can access the backup files on my Windows 10 PC without too much trouble? Furthermore, when I transfer the backup files from my PC to server B in the future, I won’t face issues such as missing symlinks etc.