Jan Harkes <jaharkes_at_cs.cmu.edu> writes:

> On Wed, Feb 22, 2006 at 10:07:53AM -0500, Greg Troxel wrote:
> > My server rebooted ungracefully (due to the xen dom0 below it
> > rebooting), and now codasrv won't start up.  Srv{Log,Err} follow.
> > Find of /usr (where /vice and /vicepa are) -inum 1562 and 1563 both
> > turn up empty.
> 
> The inodes numbers it is complaining about are most likely not the
> on-disk inode numbers of the container files, but are mapped to the
> names depending on the fanout settings of the ftreedb (i.e. 255 = 0/0/f/f)
>
> The log messages are probably still remnants from the time Coda used a
> dedicated partition to store container files instead of a filesystem
> tree and directly accessed container files with some userspace
> implementation of iget/vget.

So the log message should be fixed, or perhaps inoder should print out
the container file name.

The RT bug tracker on the web site seems non-functional (I couldn't
view the existing tickets with 'open tickets', and search got be a
perl stack backtrace.)  What's the canonical place to record bugs like this?

> I think this is why there is a 'inoder' tool on the servers,
> 
>     # inoder /vice/db/vicetab /vicepa header 50
>     Partition /vicepa: inodes in use: 5466, total: 16777216.
> 
>     Date: Wed 02/22/2006
> 
>     14:57:24 Partition /vicepa: 19593852K available (minfree=5%),
>     19429384K free.
>     Header for inode 50
>     lnk   1
>     vol   1000001
>     vnode 140
>     uniq  374
>     vers  1
>     magic 47114711
> 
> With this you can increment or decrement the linkcount of an object, it
> doesn't look like it actually tells us what the path to the file in
> question is.
 
I did find inode 1562 with a link count of 1, and 1563 was apparently
unallocated.

> What I think happened was that a file was in the process of being
> added/updated while the server went down. The container file was
> created, but the server crashed before the RVM transaction was
> committed.
> 
> Then during recovery the incomplete RVM transaction was aborted, and
> during salvage the server noticed there was a file that didn't have a
> corresponding on-disk container file. It tries to recreate the container
> file, which conflicts with the one that already exists.
 
Ah, so container file changes need to be part of RVM, but that's hard.
Perhaps a new container file needs to be allocated on each update with
a RVM transaction to point the fid at it, or something like that.

> I'm not sure why it complains about bitmap 1562 and inode 1563, maybe
> bitmap bits are counted from 0, container file numbers counted starting
> from 1.

more RT fodder...

> If you know the ftreedb parameters (width,depth) it is possible to map
> from the inode to the actual location.

My system is width 32, depth 4 and I was able to do this.

> If the server is trying to create an empty file container removing the
> container file should fix the problem. If the fso happens to be a
> directory it might crash later on because the newly created object
> doesn't have '.' and '..' entries and such.

I"m back up - I guess we'll see.

I did decrement an inode, but apparently I should not have.  I suspect
I should reinit my server, but perhaps I can first add a second
replicated server, take this one out and add it back to avoid having
to restore.

Or can backup/restore preserve acls and mod times now?

I made a wiki node about this problem:

http://coda.wikidev.net/Inconsistent_inode_errors

claude root 18 /vice/srv #> inoder /vice/db/vicetab /vicepa header 1562
Partition /vicepa: inodes in use: 4357, total: 1048576.

Date: Wed 03/01/2006

08:46:50 Partition /vicepa: 54920992K available (minfree=4%), 35680608K free.
Header for inode 1562
 lnk   1
 vol   1000002
 vnode c4e
 uniq  e5b7
 vers  2
 magic 47114711
claude root 19 /vice/srv #> inoder /vice/db/vicetab /vicepa idec 1562
Partition /vicepa: inodes in use: 4357, total: 1048576.

Date: Wed 03/01/2006

08:46:57 Partition /vicepa: 54920992K available (minfree=4%), 35680608K free.
claude root 20 /vice/srv #> inoder /vice/db/vicetab /vicepa header 1562
Partition /vicepa: inodes in use: 4356, total: 1048576.

Date: Wed 03/01/2006

08:47:03 Partition /vicepa: 54920992K available (minfree=4%), 35680624K free.
Header for inode 1562
 lnk   0
 vol   0
 vnode 0
 uniq  0
 vers  0
 magic 0




-- 
        Greg Troxel <gdt_at_ir.bbn.com>