Coda File System

Re: Performance problem

From: Jan Harkes <jaharkes_at_cs.cmu.edu>
Date: Thu, 14 Nov 2002 15:41:25 -0500
On Thu, Nov 14, 2002 at 08:00:23AM -0500, Greg Troxel wrote:
> Hint: it would be really cool if venus/vice would maintain
>   rpc packets transmitted
>   rpc packets ack failures
>   sftp packets transmitted
>   sftp packet ack failures
>   disconnections
> on a per-peer basis and dump these periodically to the log.
> I miss not having 'netstat -p rpc2 -s'.

We get these stats from the rpc2 layer, which isn't keeping track of it
on a per host basis. For some types of errors we can't even reliably
tell which rpc2 connection they are associated with.

> I did find this in the server log (up 14d 20h).  There seem to be lots of RPC
> invalid packets received, and I'm not sure what to make of SFTP's 105
> retires, 1837 timeouts and 13123 bogus.
> 
> 00:14:28 RPC Packets retried = 520, Invalid packets received = 30333, Busies sent = 705
> 00:14:28 RPC Requests 29893, Good Requests 28687, Replies 26857, Busies 25

Invalid packets are packets that filtered out as being corrupted on a
low level in the socketlistener thread. Wrong rpc2 version stamp,
truncated packets, packets that arrive after the waiting thread has
already timed out, incorrect sequence number, etc.

Typical ones I've seen are wrong rpc2 version, probably old clients that
someone forgot about and are still automatically started when the
machine is rebooted. Packets that arrive after the timeout expired, and
in some cases truncated packets, most of these were traced back to
misbehaving switches/routers or buggy network drivers.

Setting the rpc2 debug level to 10 shows the error messages, as well as
a lot of other junk). Having more than 50% of the received rpc2 packets
dropped doesn't sound too good, and I just looked at the stats of one of
our servers. Still a lot of invalid packets, but it is less than 10%...

13:18:35 RPC Packets retried = 2125, Invalid packets received = 788114, Busies sent = 2828
13:18:35 RPC Requests 9084291, Good Requests 9032774, Replies 369691, Busies 119


I would expect SFTP timeouts to be relatively high. I believe it is
bumped whenever the transfer stalls, which as I recall happen a lot when
a client is writing to a server. (writes are typically a lot slower than
reads)

Jan
Received on 2002-11-14 15:43:58