(Illustration by Gaich Muramatsu)
On Sat, May 19, 2007 at 12:00:35AM +0200, Enrico Weigelt wrote: > * shivers_at_ccs.neu.edu <shivers_at_ccs.neu.edu> wrote: > > Note that if coda were rock solid and scaled like this, > > then you would never have to waste disk space setting up > > RAID arrays on every disk system in your life -- you'd > > only need to RAID your coda *server*, while all your > > client boxes could run un-RAIDed. > > AFAIK, if you run multiple mirrored servers, you probably can > even live w/o RAID. If one system fails, another one jumps in. Right, our servers don't use RAID. If one server fails we just keep running with a smaller available volume group. The only problem is that resolution logs don't get truncated until the missing server returns, so the remaining server(s) will at some point run out of resolution log space and trigger an assert. > But I didn't test this yet. Probably depends on Coda's behaviour > on broken storage. > > So the big question is: what happens if the an underlying FS fails ? > Does Coda properly detect this and drive to some clean state > (ie. take an damaged volume offline and kick clients to the > next server ?) It depends on the failure. If the underlying FS just silently corrupts data then the files from that server are corrupted. Some client may never even consider that server as the best connected so they wouldn't even see any problems. If the failure hits either RVM, or makes things fail with IO errors, then the server tends to hit an assertion and as a result become unreachable. Most of the failures I've seen with our servers are really hard-stop type. We've had unexpected shutdowns when a UPS failed, a powersupply that broke, once the CPU fan must have fallen off and a server went into thermal shutdown. And then of course the occasional harddrive that just refuses to spin up after a reboot. I've replaced failed servers by installing an empty Coda server on a new machine and giving it the same ip-address. Then with a list of all the volume replicas that used to exist on the dead machine I created identical empty volumes. The only thing needed then is to tell a client to recursively walk the tree. It starts at the root and notices that the new server has an all zero version vector, so it copies the root directory from another replica, then it creates empty files for all directory entries and as such we resolve/repopulate all volumes. At first there will be a lot of errors in the logs from other clients that are trying to access files that do not yet exist on the new server, sometimes I set up a firewall to block out all other clients until I've resolved at least all directories. I've also used this when we upgraded the Coda servers to new hardware, bringing up the new machines for some replicas, resolving until they were in sync, and then replaced the remaining replicas and resolved again. It takes a while, but avoided the need to go back to backups. JanReceived on 2007-05-18 23:42:10