(Illustration by Gaich Muramatsu)
(Illustration by Gaich Muramatsu)
The MultiRPC facility < @@ref > Satyanarayanan90[Satyanarayanan 90] is an extension to RPC2 that provides a parallel RPC capability for sending a single request to multiple servers and awaiting their individual responses. Although the actual transmission is done sequentially, the resultant concurrent processing by the servers results in a significant increase in time and efficiency over a sequence of standard RPC calls. The RPC2 runtime overhead is also reduced as the number of servers increases. For the purposes of this discussion, the base RPC2 facility will be referred to simply as RPC2 .
A noteworthy feature of the MultiRPC design is the fact that the entire implementation is contained on the client side of the RPC2 code. The packet which is finally transmitted to the server is identical to a packet generated by an RPC2 call, and the MultiRPC protocol requires only a normal response from a server.
A major design goal was the desire to automatically provide MultiRPC capability for any subsystem without requiring any additional support from the subsytem designer or implementor. This has been achieved through modifications to RP2Gen, the RPC2 stub generation package (see chapter < @@ref > Satyanarayanan90[Satyanarayanan 90] < @@ref > RP2GenRP2Gen). RP2Gen generates an array of argument descriptor structures for each server operation in the specification file, and these arrays are inserted in the beginning of the client side stub file. These structures are made available to the client through definitions in the associated .h file, and allow the use of MultiRPC with any routine in any subsystem with RP2Gen generated interfaces.
The orthogonality of the MultiRPC modifications also extends to the side effect mechanism. Side effects for MultiRPC work exactly as in the RPC2 case except that the client must supply a separate SE_Descriptor for each connection. Side effects can be omitted on a subset of the connections by specifying a the Tag field value of OMITSE for those SE_Descriptor s. This used, for example, in the Coda file system where a MultiRPC call is made to obtain file version information from a set of servers, but data from only one of those servers.
Parameter packing and unpacking for MultiRPC is provided in the RPC2 runtime library by a pair of routines. These library routines provide the functionality of the client side interface generated by RP2Gen as well as some additional modifications to support MultiRPC. It was decided to perform the packing and unpacking in RPC2 library routines rather than in individual client side stub routines as in the RPC2 case; this requires some extra processing time, but saves a significant amount of space in the client executable file. This approach has the added advantage of modularity; execution of RPC2 calls will not be affected at all, and even for MultiRPC calls the additional processing time is negligable in comparison to the message transmission overheads imposed by the UNIX kernel.
Another feature of MultiRPC is the client supplied handler routine. Through the handler routine the client is allowed to process each server response as it arrives rather than waiting for the entire MultiRPC call to complete. After processing each response, the client can decide whether to continue accepting server responses or whether to abort the remainder of the call. This facility can be useful if only a subset of responses are required, or if one failed message renders the entire call useless to the client. This capability is discussed further in section XXX .
MultiRPC also provides the same correctness guarantees as RPC2 except in the case where the client exercises his right to terminate the call. RPC2 guarantees that a request (or response) will be processed exactly once in the absence of network and machine crashes; otherwise, it guarantees that it will be processed at most once. If the call completes normally, a return code of RPC2_Success guarantees that all messages have been received by the appropriate servers.
To illustrate the use of MultiRPC, we revisit Example XXX from Chapter XXX and Example XXX from Chapter XXX . The interface specification files ( rtime.rpc and rcat.rpc ) and the files for the server code ( rtime_srv.c and rcat_srv.c ) remain unchanged. Only the client files ( rtime_clnt.c and rcat_lowbar;clnt.c ) have to be modified.
This example specifies no handler, and does not perform any side effects. The client will prompt for the number of servers to which the request is to be made, and for their connection ids. Note that RPC2_MultiRPC is used even when only one server is requested.
@include(multi_rtime_clnt.c.mss)
In this example, a minimal handler routine is supplied for each server operation. It is adequate to demonstrate the format of the routine even though it does little actual processing of the responses. This example also demonstrates the use of MultiRPC in conjunction with side effects.
@include(multi_rcat_clnt.c.mss)
The following table shows the C type interface between the client routine and MRPC_MakeMulti for all the possible combinations of legal parameter declarations and types. In all cases it is the clients responsibility to allocate storage for all parameters, just as in the RPC2 case. For all types except RPC2_EncryptionKey, IN parameters are handled the same as in the single MakeRPC case; RPC2_EncryptionKeys must be passed as pointers in the MultiRPC case. For OUT and INOUT parameters, arrays of pointers to parameters must be supplied in order to hold the multiple server responses. The array for each parameter must contain the same number of items as the number of servers contacted, and they must be filled sequentially starting from element zero. For all INOUT parameters except for SE_Descriptors , only the first element of the array need be filled in. For SE_Descriptors , all elements must be filled in.
The following table should be consulted for specific formats.
|
RPC2 Type |
in | out | in out > )RPC2_Integer | long | long *[ ] | long *[ ] |
| RPC2_Unsigned | unsigned long | unsigned long *[ ] | unsigned long *[ ] | |||
| RPC2_Byte | unsigned char | unsigned char *[ ] | unsigned char *[ ] | |||
| RPC2_String | unsigned char * | unsigned char **[ ] | unsigned char **[ ] | |||
| RPC2_CountedBS | RPC2_CountedBS) * | RPC2_CountedBS) *[ ] | RPC2_CountedBS) *[ ] | |||
| RPC2_BoundedBS | RPC2_BoundedBS) * | RPC2_BoundedBS) *[ ] | RPC2_BoundedBS) *[ ] | |||
| RPC2_EncryptionKey | RPC2_EncryptionKey * | RPC2_EncryptionKey *[ ] | RPC2_EncryptionKey *[ ] | |||
| SE_Descriptor | illegal | illegal | SE_Descriptor *[ ] | |||
| RPC2_Enum name | name | name *[ ] | name *[ ] | |||
| RPC2_Struct name | name * | name *[ ] | name *[ ] | |||
| RPC2_Byte name[...] | name | name *[ ] | name *[ ] | |||
The client is only responsible for understanding the parameter type interface to the MakeMulti and HandleResult routines, and for allocating all necessary storage. MRPC_MakeMulti and MRPC_UnpackMulti are included in the RPC2 libraries.
@newpage
MRPC_MakeMulti ( in long Server in ARG ArgTypes[], in long HowMany, in RPC2_Handle CIDList[], out RPC2_Integer RCList[], in out RPC2_Multicast *MCast, in long (*HandleResult) (), in struct timeval *Timeout, Variable Length Argument List[] )
For server routine foo, "foo". RP2GEN generated opcode, defined in include file. Note that subsystems with overlapping routine names may cause problems in a MakeMulti call.,
For server routine foo, "foo;PTR". RP2GEN generated array of argument type specifiers. A pointer to this array is located in the generated include file foo.h .,
How many servers are being called,
Array of HowMany connection handles, one for each of the servers,
Array of length HowMany, into which RPC2 will place return codes for each of the connections specified in ConnHanleList. May be specified as NULL if return codes will not be examined.,
Pointer to multicast sturcture. Set to NULL for now.,
User procedure to be called after each server response. Responses are processed as they come in. Client can indicate when he has received sufficient responses (see below). @MRPC(MakeMulti) will not return the server responses.,
User specified timeout. Note that the default timeout set in the .rpc file will not be active here: a NULL value will be passed through to MultiRPC, where it will indicate infinite patience as long as RPC2 believes that the server is alive. Note that this timeout value is orthogonal to the RPC2 internal timeout for determining connection death.,
This is just the list of the server arguments as they are declared in the .rpc2 file. It is represented in this form since each call will have a different argument list.,
All went well,
The user specified timeout expired before all the server responses were received,
Something other than SUCCESS or TIMEOUT occurred. Individual server response is supplied via UnpackMulti to the client handler routine.
Text=`For all in or in out parameters, an array of HowMany of the appropriate type should be allocated and supplied by the client. For example, if one argument is an out integer, an array of HowMany integers (i.e. int foo[HowMany]) should be used. For structures, an array of structures and NOT an array of pointers to structures should be used. in arguments are treated as in the @RPC2(MakeRPC) case. )
RPC2_MultiRPC( in long HowMany in RPC2_Handle ConnHandleList[], in RPC2_PacketBuffer *Request, in SE_Descriptor SDescList[], in long (*UnpackMulti) (), in out @ARG(INFO) *ArgInfo, in struct timeval *Patience )
How many servers to contact,
List of HowMany connection handles for the connections on which calls are to be made.,
A properly formatted request buffer.,
List of HowMany side effect descriptors,
Pointer to unpacking routine called by RPC2 when each server response as received. If RP2Gen is used, this will be supplied by @MRPC(MakeMulti). Otherwise, it must be supplied by the client.,
A pointer to a structure containing argument information. This structure is not examined by RPC2; it is passed untouched to UnpackMulti. If RP2Gen is used, this structure will be supplied by @MRPC(MakeMulti). Otherwise, it can be used to pass any structure desired by the client or supplied as NULL.,
Maximum time to wait for remote sites to respond. A NULL pointer indicates infinite patience as long as RPC2 believes that the server is alive. Note that this timeout value is orthogonal to the RPC2 internal timeout for determining connection death.,
All servers returned successfully, or all servers until client-initiated abort returned successfully. Individual server response information is supplied via UnpackMulti to the user handler routine supplied in the ArgInfo structure.,
The user specified timeout expired before all the servers responded.,
Something other than SUCCESS or TIMEOUT occurred. More detailed information is supplied via UnpackMulti to the user handler routine supplied in the ArgInfo structure.,
Logically identical to iterating through ConnHandleList and making RPC2_MakeRPC) calls to each specified connection using Request as the request block, but this call will be considerably faster than explicit iteration. The calling lightweight process blocks until either the client requests that the call abort or one of the following is true about each of the connections specified in ConnHandleList: a reply has been received, a hard error has been detected for that connection, or the specified timeout has elapsed.
The ArgInfo structure exists to supply argument packing and unpacking information in the case where RP2Gen is used. Since its value is not examined by RPC2, it can contain any pointer that a non-RP2Gen generated client wishes to supply.
Similarly, UnpackMulti will point to a specific unpacking routine in the RP2Gen case. If the RP2Gen interface is not used, you should assume that the return codes of the supplied routine must conform to the specifications in section < @@ref > UnpackMultiXXX.
Side effects are supported as in the RPC2 case except that the client must supply a separate @SE[Descriptor] for each connection. The format for the @SE[Descriptor] argument is described in section XXX . It will often be useful to supply connection specific information such as unique file names in the @SE[Descriptor] .
MRPC_UnpackMulti( in long HowMany, in RPC2_Handle ConnHandleList[], in out @ARG(INFO) *ArgInfo, in RPC2_PacketBuffer *Response, in long rpcval, in long )
How many servers were included in the MultiRPC call,
Array of HowMany connection ids,
Pointer to argument information structure. This pointer is the same one passed in to MultiRPC, so for the non-RP2Gen case its type is determined by the client.,
RPC2 response buffer,
Individual connection error code or server response code,
Index into ConnHandleList to identify the returning connection,
Continue accepting and processing server responses,
Abort MultiRPC call and return,
This routine is fixed in the RP2Gen case, and can be ignored by the client. For the non-RP2Gen case, a pointer to a routine with the argument structure described must be supplied as an argument to RPC2_MultiRPC). The functionality of such a client-supplied routine is unconstrained, but note that the return codes have an important effect on the process of the MultiRPC call.
HandleResult( in long HowMany, in RPC2_Handle ConnArray[], in long WhichHost, in long rpcval, [Variable Length Argument List] )
number of servers from @MRPC(MakeMulti) call,
array of connection ids as supplied to @MRPC(MakeMulti),
this is an offset into ConnArray and into any OUT or IN;OUT parameters. Using this to index the arrays will yield the responding server and its corresponding argument values.,
this is the RPC2 return code from the specified server, p5=` < Variable Length Argument L
Continue processing server responses,
Terminate @MRPC(MakeMulti) call and return,
This routine must return either 0 or -1. A return value of zero indicates that the client wants to continue receiving server responses as they come in (normal case). A return value of -1 indicates that the client has received enough responses and wants to terminate the MakeMulti call (in which the client is still blocked). This allows the client to call a large number or servers and terminate after the first n responses are received.
Note that the name of this routine is arbitrary and may be determined by the client. RPC2_MultiRPC) sees it only as a pointer supplied as an argument to @MRPC(MakeMulti). The parameter list is predefined, however, and the client must follow the structure specified here in writing the routine.