Hello!

Thank you Jan for your answer.

> -----Original Message-----
> From: Jan Harkes [mailto:jaharkes_at_cs.cmu.edu]
> Sent: Tuesday, March 13, 2007 4:57 PM
> To: Florin Muresan
> Subject: Re: venus crash
> 
> On Tue, Mar 13, 2007 at 12:45:32PM +0200, Florin Muresan wrote:
> > Hello everybody!
> 
> I guess this was intended to be sent to codalist,

Yes it was but I missed the reply all button. Sorry for that.

> 
> > I have a similar problem in my coda realm. Venus crashes when I try
to
> > copy(overwrite)/delete many files.
> > I use coda in a production environment for a web hosting solution.
> > During the testing period this situation never happened. My guess is
> > that the problem occurs because of the high number of accessed files
per
> > second that triggers false conflicts.
> 
> High rate of accesses isn't really something that would trigger false
> conflicts. Also, false conflicts should not cause a client to crash.

I must say that you are right and I had made a mistake saying that Coda
client crashes. In fact it just hangs. I had read some of the emails
from the codalist posted last year and I think I understand better how
Coda works.

> 
> There are some problems that I have observed with our own web/ftp
> servers, which get pretty much everything out of /coda.
> 
> - The Coda client is a userspace process, so all file open and close
> requests are forwarded to this single process where they are queued.
> There is a set of about 20 worker threads that pick requests off the
> queue and handle them. This is in a way similar to how a web-server
> has one incoming socket where requests are queued, which are then
> accepted by a web-server instance (thread or process).
> 
> In-kernel filesystems have it a bit easier, when the webserver makes a
> system call the kernel simply uses the application context, so there
> is no queueing and it can handle as many requests in parallel as there
> are server instances.
> 
> Now we only really see open and close requests all individual read and
> write calls are handled by the kernel, so if the client has most or
all
> files cached the worker threads don't have to do much and are pretty
> efficient. However if a some (or a lot) of files are updated on the
> server, most the locally cached data is invalidated and almost every
> request ends up having to fetch data from the Coda servers. So each
> request takes longer, and may have some locks on a volume because it
is
> updating the local cache state. So in the worst case only process a
> single request at a time and the queue becomes very long, blocking all
> web-server processes.

I think this is exactly what happened when I tried to delete about 900
files at one time and the result was that apache webserver got blocked,
slowing down the whole system. 

My Coda setup is formed from one Coda server and three Coda clients (for
the moment). It is very important to have the same web documents on all
three clients because I implemented an load-balancing solution and all
clients must serve to visitors the same content. Coda solves this very
elegantly.

Back to the point. Trying to solve the problem I terminated the apache
processes and then restarted venus. After restart the whole /coda volume
was in disconnected state and I feared that I lost all the data. I had
to move quickly because all of my websites were down and the only quick
solution that I could think of was to purge and reinstall the
coda-client package on the system where I deleted the files. I thought
this way I will avoid any conflicts that could appear.
The curios thing is that the other two Coda clients were hanging after
this problem occurred.

For the sake of readability I atached the relevant logs.

After I reinstalled the coda-client and restarted the Coda server the
/coda become accessible on all the clients but I had to waith for a
while to reintegrate.

I belive now that it wasn't necesary to reinstall the coda-client
because all the conflicts would been resolved automaticaly and that an
restart of the Coda server would have been enough.

> 
> - Another thing that can happen is that when one client is updating a
> file, another client sees the update before the final commit. At this
> point the versions are still skewed. A write first updates each
replica
> and then uses the success/failure status to synchronize the versions.
> So if we see a changed object before the versions are synchronized the
> reading client believes there is a conflict and triggers server-server
> resolution. As a result the servers lock down the volume, exchange
their
> copies of the file or directory, compare the differences and decide
> which one would be considered the most up-to-date version.
> 
> We detect the missed version synchronization because the contents are
> identical, this is a 'weak-equality' type resolution and so the
servers
> reset the versions to be correct again. Then when the writing client
> finalizes the operation, the versions end up getting bumped for a
second
> time, skewing them again, requiring the reading client to refresh it's
> cache and triggering another resolution. There is not a correctness
> issue here, but the additional 2 resolution phases definitely slow
down
> everything because they add an additional 10 roundtrips and take an
> exclusive lock on the volume, preventing all readers and writers even
> for unrelated objects within the volume.
> 
> Neither of these would introduce crashes or conflicts though, mostly a
> temporary performance degradation where all web servers are blocked
> until the system catches up again with all the queued requests.

Its clear for me now why I get this performance degradation when trying
to copy many files but this is not at all desireable in an production
environment.

> 
> > Do you think that if I install the version 6.9.0 the problem with
false
> > conflicts will be avoided?
> 
> Not sure if false conflicts are your problem. A crash is a bug, even
> when there are conflicts we shouldn't crash. With 6.9 we basically end
> up using an existing code-path that was normally only used during
> disconnections or when connectivity was poor. That code has been
around
> for a long time, but really hadn't been tested all that well because
it
> was the fallback path. So I do actually expect it to be somewhat less
> reliable. However as it is the same code that is used by older clients
> when things went wrong it isn't really a step back. Any bugs that
could
> happen in unusual situations have become bugs that will happen.
> 
> > What any other suggestions for this situation?
> 
> What does your setup look like? Are there replicated volumes or is
> everything backed by a single Coda server (i.e. a single server would
> never have resolution issues). How many Coda clients are there. Are
> updates being written by one client or by several client. Which client
> crashes, the one that is writing or another that is only reading.

I had described the setup above but I must make some remarks. Typicaly,
all the Coda clients will write from time to time. One of them is mainly
used to write files, but only on specific times and was not doing any
writing at the moment of the incident. In this case, the client that
hanged was one of the clients that rarely writes. At that time I used it
to delete the files. 

> 
> What web server are you using? How many threads/processes does it use?
> How many requests per second are we talking about?

I run on every client Apache 2.0.54 with mpm_prefork_module and is
currently setup for 200 MaxClients. The prefork module uses only one
thread/process. The average of requests vary from 20 to 50 requests per
client per second.

> 
> What is logged in /var/log/coda/venus.err when the client crashes?
> 
> Jan


Thank you for your time.
Florin