THE RANT /
THE SCHPLOG
Schmorp's POD Blog a.k.a. THE RANT
a.k.a. the blog that cannot decide on a name
All Articles
⟵ 2015-07-01 Emulating Linux-MIPS in Perl - Part 3: MIPS CPU emulation
⟶ 2015-07-04 Emulating Linux-MIPS in Perl - Part 4: Bits and Ends and Links

This document was published 2015-07-02 21:55:39, and since then has not been materially modified.

Tidbits - a Linux NFS client bug

Just had a weird new NFS client bug (with Linux 4.0.1 - I'm suffering from various NFS bugs since about 2.6.18, so debugging NFS issues is kind of routine).

I don't know what triggered it, but my girlfriend complained her vi froze, and couldn't explain why (and neither could I). As it turned out, it was stuck in fsync, and since she was editing on an NFS volume, I quickly guessed it's an NFS problem, as vim was probably trying to write its swapfile.

tcpdump quickly showed the problem - the NFS client tried to connect to the server every three seconds, but was rejected:

23:28:06.188527 IP 10.0.0.230.684 > 10.0.0.5.2049: Flags [S], seq 8126
23:28:06.188576 IP 10.0.0.230.684 > 10.0.0.5.2049: Flags [S], seq 9041
23:28:06.189458 IP 10.0.0.5.2049 > 10.0.0.230.684: Flags [S.], seq 3788, ack 8127
23:28:06.189477 IP 10.0.0.230.684 > 10.0.0.5.2049: Flags [R], seq 8127, win 0
23:28:06.189926 IP 10.0.0.5.2049 > 10.0.0.230.684: Flags [R.], seq 3508, ack 916, win 0

After checking firewall, conntrack etc., I looked closer after the actual tcpdump output and realised what the problem was: the client 10.0.0.230 sent out two SYNs in very quick succession (within less than 50µs), but the SYNs were actually for two different connections!

The server 10.0.0.5 replies to the first, and probably dropped the second. The NFS client then rejected the SYN ACK from the server, presumably because it had already forgotten the first connection request.

This is the moment where I already considered rebooting, as my experience with Linux NFS problems is that you can't solve them without rebooting, as you usually can't umount or remount the mountpoint when it is stuck.

Workaround

Fortunately, it turned out to be fixable, at least temporarily:

mount -noremount,udp /path/to/mountpoint

This took a few seconds, then returned, together with all the other commands that were previously stuck (vim, sync, ls...).

I had expected it to switch the client to UDP, but it stayed with UDP, so it seems the remount alone sufficed.

I've never seen this failure mode before, so maybe it was a one time glitch. Let's wait and see...

⟵ 2015-07-01 Emulating Linux-MIPS in Perl - Part 3: MIPS CPU emulation
⟶ 2015-07-04 Emulating Linux-MIPS in Perl - Part 4: Bits and Ends and Links

About Me

I'm generally known as schmorp outside of perl, and often called by my CPAN handle mlehmann by perl people, although I don't identify with the latter.

I'm mostly known for my software, most of which you can find on software.schmorp.de or on CPAN. Most prominent are probably libev, a high performance event library, rxvt-unicode, a vt102 terminal emulator, and AnyEvent, which allows Perl module authors to write event-based code without locking them into a specific event library.

Comments

This blog does not have the typical comment entry form. There are currently two ways of commenting:

RSS feed

An experimental RSS feed is also available.

Copyright (C) 2015 Marc A. Lehmann <blog@schmorp.de>. Do not copy or excerpt substantial parts without permission unless otherwise noted. Short code snippets and example code is in the public domain.