On Mon, Jul 24, 2000 at 05:31:48PM +0200, Florin Grad wrote:
> 
> Hello again,
> 
> I have some questions about servers crash/rebuild :
> 
> Mainly I asked myself these questions from the very beginning and did find
> a clear answer yet:
> 
> 1. Coda backup coordinator
> 
>    Do we have to use the coda backup coordinator machine or any other
>    backup system will do ? We could, for example restore the entire disk
>    and mount it on /vicepa afterwards. Will that create too many
>    inconsistencies ?

/vicepa is only the data part of the files, the RVM log and data files
contain all the metadata and directory information. It is pretty much
impossible to grab a consistent snapshot without completely shutting
down the server. Coda's backup mechanism locks a volume for a short time
while the backup clone is created which can then be pulled off the
server at any later time, and as such makes a consistent snapshot of the
volume with minimal downtime.

> 2. Non-scm crash/ rebuild
> 
>    The partition of one of the non-scm servers that I had on amovible hard
>    drive crashed because of some test I did. It was on an reiserfs
>    partition. 
>    Anyway, I re-installed the non-scm server, it managed to get the
>    /vice/db files 
> 
>    Question: how do I get the volume to populate the non-scm /vicepa
>    directory. I type bld...sh non-scm server on the scm server, it said
>    success bla bla but this still doesn't work.

It doesn't really matter whether you are rebuilding an SCM or a non-SCM
server. The important part is to recreate the lost volume replicas with
the same name, replicated volume-id and volume replica-id.

The volumename of a replica is the replicated volume name + an
uniquifying number, i.e. rootvolume.0.

    volutil create_rep <partition> <replicaname> <replicated-id> <replica-id>

e.g.

    volutil create_rep /vicepa volume.1 7F000010 2000007

You should be able to pull the required numbers out of the VRList on the
SCM, which contains almost all needed information. According to the logs
the restored server has server-id 2, so any volume starting with
`200....' would have to be recreated. The position in the VRList is a
good indicator of which uniquifying number should be tagged after the
replicated volume name.

So let's assume I want to recreate lost volumes on server `e5' 

    # grep e500 /vice/vol/VRList
    e:u.jaharkes.tallis 7F0004AB 1 e5000021 0 0 0 0 0 0 0 E0000153
    e:jaharkes.rep 7F0004C5 2 e60000f4 e5000024 0 0 0 0 0 0 E0000149
    e:u.jaharkes.tt 7F00053A 2 e60000fd e5000001 0 0 0 0 0 0 E0000149
    reptest 7F000549 2 e6000101 e5000002 0 0 0 0 0 0 E0000149
    e:u.rim.rep 7F00054B 2 e6000102 e5000003 0 0 0 0 0 0 E0000149

So I need the following operations to recreate the volumes on the
restored server.

    volutil create_rep /vicepa e:u.jaharkes.tallis.0 7F0004AB e5000021
    volutil create_rep /vicepa e:u.jaharkes.rep.1 7F0004C5 e5000024
    volutil create_rep /vicepa e:u.jaharkes.tt.1 7F00053A e5000001
    volutil create_rep /vicepa reptest.1 7F000549 e5000002
    volutil create_rep /vicepa e:u.rim.rep.1 7F00054B e5000003

And now life becomes simple, rerun `bldvldb.sh' on the SCM and simply do
ls -lR in each `restored volume'. This will trigger resolution and the
empty replica will get the lost data back.

>    So how do I add a new non-scm server on a coda cell an dpopulate it
>    with the existing volumes, I should modify the /vice/db/VSGDB adn type
>    bldvldb.sh new-non-scm again ? But how will the others non-scm servers
>    know about it ?

Expanding (or shrinking) existing volumes to completely new servers
currently does not work without reinitializing all clients. The client
needs some more cleanups in the code that handles volumes before this
can be done.

There is really no difference between SCM and non-SCM servers except for
the fact that non-SCM servers fetch updated /vice/db files from the SCM,
so that the `read-write' copy has to be on the SCM.

>    If I try to connect the client I get the error message:
> 
> cd /coda/pub
> 17:26:38 MaxRetries exceeded...returning EWOULDBLOCK
> 17:26:45 MaxRetries exceeded...returning EWOULDBLOCK
> bash: cd: /coda/pub: Ressource not available or something like that

The client tries to resolve the differences, but fails.

> on the non-scm server I have in /vice/srv/SrvLog
...
> 15:21:35 GetVolObj: VGetVolume(2000007) error 103

Because the volumes don't exist on the recovered server.

> 3. Scm crash/rebuild
> 
>    Now, if the scm crashes or if I turn it off, will the cell still be
>    working, eventually i I restart the clients ?

Yes, there should be no problem, the only file which is at the moment
`unique' to the SCM is /vice/vol/VRList, and we'll probably move that
one to /vice/db in the near future. Once that file is available on all
servers, we can elect any other available server as the new SCM by
putting it's name in the /vice/db/scm file on all servers and restarting
the updateclnt/updatesrv processes.

Here we have our client configured to use 3 of the production servers as
our `rootservers', so even when 2 of them are unavailable, the clients
will still be able to get volume information, i.e. only the rootservers
are queried to discover where a volume is located.

Jan