On Mon, Nov 26, 2018 at 10:28:34AM -0500, Greg Troxel wrote:
> Jan Harkes <jaharkes_at_cs.cmu.edu> writes:
> 
> > We used a modified Andrew benchmark to test by untarring and building a
> > Coda source tree, locally, in Coda using the Coda kernel module, and
> > using the Linux 9pfs kernel module with and without caching.
> >
> > - on local filesystem,               1m 59s
> > - in Coda using Coda kernel module,  2m  9s
> > - in Coda using 9p2000,             67m 39s
> > - in Coda using 9p2000 with caching  3m 51s
> > - in Coda using 9p2000.u,           70m 23s
> > - in Coda using 9p2000.L,           ~5m
>
> Do you understand where the time is going?  A factor of 2 or 3 does not
> seem surprising, but 30x is.

Well the total amount of dirty state is pretty limited when all you do
is compile Coda. So in the local filesystem and Coda kernel module case
we are probably mostly measuring compilation time. Normal reads and
writes go directly to the pagecache and if there are no fsyncs we may
never even have to wait for the disk.

Once we go through v9fs, every single read and write operation becomes
quite a bit less efficient.

    - kernel sends request to Venus
    - Venus maps from plan9 qid to Coda fid and looks up the fsobj
    - perform the read (or write) operation
        - open the container file
        - seek to offset, read/write data
        - close container
    - Venus sends response to kernel

As far as I remember, we don't try to cache the open file handles so
writes probably end up triggering some I/O to update the on-disk inode
every time the file descriptor is closed, which is every write call. And
small or misaligned writes would result in multiple read-modify-write
cycles. If we kept an open container file handle cache in Venus some if
this would be more efficient but Venus had never before needed such a
cache because it normally doesn't see operations between open and close.

So I'm not completely surprised about 30x overhead, especially when
there are a lot of writes. The amount of data is small enough that
technically all reads could come from the pagecache, not sure if that
can happen here though.

Jan