On Mon, 27 Jul 1998, Shafeeq Sinnamohideen wrote:

> 2) RPC2 for Pthreads. Right now, RPC2 is built on top of LWP and consists
> of various functions called by a thread that wants to do an RPC, and the
> SocketListener, which is an LWP that listens for incoming packets and uses
> LWP signals to hand that packet off to a worker thread that can process
> it. The SL should itself become a Pthread, for reasons metioned before. It
> should also fork new threads to handle new bind requests, since binds
> involve several packets, this is the easiest way to save state between
> packets. 
> Since a LWP condition variable will be a wrapper for a Pthread condition
> variable, it should not be too hard to handle both plain Pthreads that
> want to make RPCs, and the Pthreads_that_behave_like_LWPs from part 1 hat
> are also making RPCs. For all current users of RPC2, there should not be
> any concurrency problems yet, but performance should improve.

Replacing the IOMGR LWP package should be a relatively easy step
supporting this move -- it would not even require that we change much (if
any) code in the SocketListener initially -- IOMGR_Select would continue
to provide identical functionality, only use the libc threadsafe select
call.  Changes could be trickled up through these various packages as you
describe above and below.

One intermediate step is adding support for the LWP_Signal capability in
LWP -- presumably quite similar to the conditional support in pthreads,
and so easy to emulate.  Similarly, the support for catching unix Signals
with individual threads should carry over, as with the timer behavior (all
of these are areas to hit on the way through).

> 3) Move tasks out of LWPs into preemptive threads. Since a mix of
> preemptive and cooperative threads is supported now, tasks that are
> CPU-intensive can have protections added and scheduled preemptively. I
> beleive that RVM already works with pthreads, and I can't think of
> anything cpu-intensive that isn't RVM related or encryption. IO
> intensive threads should'nt benefit much over their current performance,
> right? 

Worker threads associated with kernel threads blocked on a kernel event
wait (possibly on multiple processers) will be able to wake up with lower
latency on an event occuring than waiting for user-land thread package to
wake up the appropriate thread (possibly after preempting).  I think this
is the advantage that Michael was hinting at.

Also, this would allow timed events to happen with greater precision when
bound to kernel threads on SMP machines, I believe.  Certainly on BSD (and
I assume pretty much all other platforms), the kernel preserves page
coloring information to maximize locality for processes -- using kernel
threads would allow us to take advantage of that also.  This is the usual
set of arguments for OS-provided thread packages over third-party ones, I
believe?

> Encryption/compression can be paralellized very easily since it is done
> right before the packet is sent, on a SMP system, we can let another LWP 
> run while we are encrypting, since we know the next thing we will
> do is send the packet, at which point the next LWP will get to run, so why
> not start it early ? IF the critical sections in RPC2 are adequately
> protected, this should be ok.

As I mentioned before (and I believe that the documentation in the
preemption or rpc2 packages describes), RPC2 is currently covered by a
single lock covering pretty much all entry points into RPC2.  As a result,
we'll have to figure out the lock push-down, and this may require some
re-architecting of some portions of RPC2.

> While this whole thing is more ugly than just replacing LWP, it delivers
> most of the benefits without a hideous amount of work.

This sounds like an excellent plan.  It also maintains the highest level
of stability as the changes are introduced, which is clearly a goal with
Coda so as to move it into production environments on the "sooner" rather
than "later" schedule :).

  Robert N Watson 

Carnegie Mellon University            http://www.cmu.edu/
TIS Labs at Network Associates, Inc.  http://www.tis.com/
SafePort Network Services             http://www.safeport.com/
robert@fledge.watson.org              http://www.watson.org/~robert/