ssh – “message authentication code incorrect” error with scp

I have an intermittent scp failures from one specific computer (Mac OS 11.2) to one specific host (Ubuntu 18.04.5). The computer has no issues with scp to other hosts, and other computers have no issues with scp to this host. I also have no issues with ssh or mosh.

On the client, I see:

$ /usr/bin/scp vegan-choux-deflated-big.jpg ps:jtk/
stty: 'standard input': Inappropriate ioctl for device
vegan-choux-deflated-big.jpg                       0%    0     0.0KB/s   --:-- ETAclient_loop: send disconnect: Broken pipe
lost connection

On the server I see:

$ sudo journalctl -f
Feb 20 19:12:08 jefftk.com sshd(15856): Accepted publickey for jefftk from 146.115.48.13 port 65263 ssh2: RSA SHA256:FT44f1oAdXEJtBZTFf1zxC2r6ZxptSES3ZkG/fPmYuk
Feb 20 19:12:08 jefftk.com sshd(15856): pam_unix(sshd:session): session opened for user jefftk by (uid=0)
Feb 20 19:12:08 jefftk.com systemd-logind(3157): New session 32 of user jefftk.
Feb 20 19:12:08 jefftk.com systemd(1): Started Session 32 of user jefftk.
Feb 20 19:12:10 jefftk.com sshd(16050): ssh_dispatch_run_fatal: Connection from user jefftk 146.115.48.13 port 65263: message authentication code incorrect
Feb 20 19:12:10 jefftk.com sshd(15856): pam_unix(sshd:session): session closed for user jefftk
Feb 20 19:12:10 jefftk.com systemd-logind(3157): Removed session 32.

It seems like message authentication code incorrect is the issue?

Manually specifying a MAC (-o MACs=hmac-sha2-512) doesn’t seem to have an effect.

This error is intermittent: often everything works correctly.

Verbose upload output:

$ /usr/bin/scp -v vegan-choux-deflated-big.jpg ps:jtk/
Executing: program /usr/bin/ssh host ps, user (unspecified), command scp -v -t jtk/
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/jefftk/.ssh/config
debug1: /Users/jefftk/.ssh/config line 25: Applying options for ps
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 74: Applying options for *
debug1: hostname canonicalisation enabled, will re-parse configuration
debug1: re-parsing configuration
debug1: Reading configuration data /Users/jefftk/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 60: Applying options for *.com
debug1: /etc/ssh/ssh_config line 64: Applying options for *.*
debug1: /etc/ssh/ssh_config line 74: Applying options for *
debug1: Connecting to www.jefftk.com (163.172.164.150) port 22.
debug1: Connection established.
debug1: identity file /Users/jefftk/.ssh/id_rsa type 0
debug1: identity file /Users/jefftk/.ssh/id_rsa-cert type -1
debug1: identity file /Users/jefftk/.ssh/localhost/id_rsa type -1
debug1: identity file /Users/jefftk/.ssh/localhost/id_rsa-cert type -1
debug1: identity file /Users/jefftk/.ssh/clusterhost/id_rsa type -1
debug1: identity file /Users/jefftk/.ssh/clusterhost/id_rsa-cert type -1
debug1: identity file /Users/jefftk/.ssh/id_ed25519 type -1
debug1: identity file /Users/jefftk/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/jefftk/.ssh/id_ecdsa type -1
debug1: identity file /Users/jefftk/.ssh/id_ecdsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to www.jefftk.com:22 as 'jefftk'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bXmp8R4JOzinavMiXjgpzJk7mjhNiPOQ61NChWaXrDo
debug1: Host 'www.jefftk.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/jefftk/.ssh/known_hosts:115
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: publickey ECDSA SHA256:7PPjwtPBwuO+UnzB/Myo/La/ptAQ5EI8YeDoHkpO7wM agent
debug1: Will attempt key: corp/normal ECDSA-CERT SHA256:X7GzO8/fLB5iiNYLqDU/lRz2I7CajWm8WJcKnwv0WnA agent
debug1: Will attempt key: /Users/jefftk/.ssh/id_rsa RSA SHA256:FT44f1oAdXEJtBZTFf1zxC2r6ZxptSES3ZkG/fPmYuk
debug1: Will attempt key: /Users/jefftk/.ssh/localhost/id_rsa 
debug1: Will attempt key: /Users/jefftk/.ssh/clusterhost/id_rsa 
debug1: Will attempt key: /Users/jefftk/.ssh/id_ed25519 
debug1: Will attempt key: /Users/jefftk/.ssh/id_ecdsa 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: publickey ECDSA SHA256:7PPjwtPBwuO+UnzB/Myo/La/ptAQ5EI8YeDoHkpO7wM agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: corp/normal ECDSA-CERT SHA256:X7GzO8/fLB5iiNYLqDU/lRz2I7CajWm8WJcKnwv0WnA agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/jefftk/.ssh/id_rsa RSA SHA256:FT44f1oAdXEJtBZTFf1zxC2r6ZxptSES3ZkG/fPmYuk
debug1: Server accepts key: /Users/jefftk/.ssh/id_rsa RSA SHA256:FT44f1oAdXEJtBZTFf1zxC2r6ZxptSES3ZkG/fPmYuk
debug1: Authentication succeeded (publickey).
Authenticated to www.jefftk.com ((163.172.164.150):22).
debug1: channel 0: new (client-session)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending command: scp -v -t jtk/
stty: 'standard input': Inappropriate ioctl for device
Sending file modes: C0644 2413695 vegan-choux-deflated-big.jpg
Sink: C0644 2413695 vegan-choux-deflated-big.jpg
vegan-choux-deflated-big.jpg                   99% 2336KB   4.8MB/s   00:00 ETAclient_loop: send disconnect: Broken pipe
lost connection