Coda File System

Re: venus choking on large directories?

From: Jan Harkes <jaharkes_at_cs.cmu.edu>
Date: Wed, 16 May 2001 16:30:52 -0400
On Wed, May 16, 2001 at 08:56:24AM -0400, Greg Troxel wrote:
> I'm trying to mirror internet-drafts into coda.  This results in a
> directory with lots of entries.

Didn't you have off-by-one errors in this part of the code before?

> The server had died too:
> 
> 08:10:29 Create returns 27 on draft-ietf-aaa-diameter-e2e-sec-01.txt (0x1000004.0x438c.0x2d7e)

That's EFBIG, i.e. there are no consecutive 'blobs' to fill the
directory entry into the directory. Looking at the name length, it would
need to try and allocate 2 blobs. I would gather that the directory is
stretched to it's limit (DIR_MAXPAGES * DIR_PAGESIZE) which is only
256KB.

> I restarted venus and codasrv, and they both started up ok.
> Then:
> 
> gdt 27 /coda/project/rfc/internet-drafts > ls|wc
>     3943    3947  135415
> 
> This seems suspiciously close to 4096.
> 
> Any clues?  I seem to be exceeeding some (arbitrary and too low!)
> limit, but I don't really know.

One blob is about 32 bytes, and we pack both the name with trailing \0
and part of the FID. So with these long names (38 characters) + null
char + 8 bytes for the fid + other overhead (4 bytes) there is a
relatively high lossage. Only 51 out of 64 bytes are used. Also, every
page has one blob reserved for the pageheader, so there is an odd number
of blobs per page, i.e. we lose 2 blobs per page when the directory only
contains long names.

128 pages * 62 blobs / 2 blobs per name = 3968 entries.
3968 entries * 38 characters = 150784 bytes

These numbers come close enough to the result of your 'ls|wc' numbers.
It would be nice if we packed the directories more efficiently, and
allowed them to grow beyond 256KB. But a lot of the cruft in this area
is legacy to avoid breaking backwards compatibility with the way servers
have been storing their directory information in RVM.

Jan
Received on 2001-05-16 16:31:12