u+codalist-p4pg_at_chalmers.se writes:

> Thanks Greg and Jan,
>
> I appreciate your suggestions ad explanations.
>
> On Tue, Jul 10, 2007 at 08:27:23PM -0400, Greg Troxel wrote:
>>   I guess that only a minor change in the includes is needed
>>   to be able to cross-compile the client on ia32 to work on amd64.
>> 
>> This sounds broken.  I think you should either build it native on amd64,
>> or build it for i386, against the standard i386 includes.  Having it
>> built for i386 but with some includes that are different from
>> i386-on-amd64 sounds like a path to madness and futher inconsistencies.
>
> That's why I tried to explain the rationale. From my perspective
> it sounds broken to maintain a separate development environment
> for the sake of extremely few programs, without any functional gain.

There are multiple issues lurking here, and it wasn't your desire that I
found broken - I thought you were trying to modify include files for the
sole purpose of building linux/i386 binaries to run on linux/amd64,
without making the changes permanant and in all versions.

So, there are several things that would need to happen to solve your
problem.  I think there's too much work, and too little need, and that
it's not going to happen, but:

1) Change the protocol among all the userland programs (via pioctl) to
have a wire-format architecture- and system-independent specification.
This means using fixed-width types all the time, and not using any
system typedefs.  That in turn will require care in packing/unpacking
the data structures, but maybe this is mostly coda data, not system
data, so it's not that bad.  This obviously changes the intra-codasrc
ABI, but that isn't guaranteed stable anyway and people always rebuild
everything together.

All that said, I really don't understand how one could truly want to end
up using 32-bit clog and 64-bit venus, or vice versa, given that they
more or less have to be built from the same source version now.

2) Either change the kernel/venus ABI to be architecture-independent, or
implement in the kernel 32-bit compatibility mode that speaks the 32-bit
protocol.

Coda has long been a research system, where there isn't a formal ABI
definition, and people are expected to rebuild in sync.  The user/kernel
interface has been more stable, but it's not like there is a document
that specifies it that people go to in order to determine which side is
wrong.

Both 1 and 2 are a huge amount of work, and it seems that there are very
few people who want to run 32-bit coda userland and a 64-bit kernel.

>>   Examination of the code around the mount() call did not reveal
>>   why it would/should fail.
>>   The struct passed by the call consists of two ints, which seems
>>   to have the same representation in both 32- and 64-bit mode.
>> 
>> So does venus compiled native in 64-bit mode work?  What does the mount
>> call look like in this case?
>
> It looks the same, but comes from a process running in different mode,
> via a different 64-bit syscall ABI - the kernel otherwise has the 32-bit ABI
> fully functional.

Then perhaps the 32-bit emulation mount code for coda is wrong.

> On Tue, Jul 10, 2007 at 10:05:18PM -0400, Jan Harkes wrote:
>> unlike most in-kernel users of ioctls, Coda's ioctls are
>> between programs like cfs and clog on the one side and venus on the
>> other. So you would have to not only deal with various possible
>> combinations of 32- and 64-bit versions of cfs/clog/venus. But to be
>> able to translate the kernel has to know exactly what data is being
>> passed along inside these ioctls. Currently the kernel can just treat it
>> as a binary blob, but if it has to translate it needs to know how to
>> decode and re-encode information that it really has no business knowing
>> about.
>
> The need for reencoding is due to the interface being defined in terms
> of architecture-dependent data structures. I see that redefining
> the interface in an architecture-independent way would break compatibility
> with the current implementation. On the other side, as long as both
> "peers" like venus-clog or venus-repair are compiled from the same code,
> it should work - if the kernel really passes data as binary blobs.

Perhaps, but that seems like asking for trouble.

> It feels otherwise frustrating to be unable to run the same code
> using an _available_ ABI, having to recompile for a specific ABI instead.

Sure.  Perhaps you have a spare $200K to send to CMU to task them to
implement the changes you desire?  (Joking, but what you are asking for
is a lot of work.)

> x86 <-> amd64 is not the only example. Linux ABI is available on
> several *BSD kernels, what is wrong with providing a single client userspace
> for all those systems? Our Coda client is self-contained and does not
> depend on any additional libraries, why do we have to maintain 5 instanses
> of it - while one x86-Linux instance is capable of running on all 5 platforms
> unmodified?..

But it isn't.  Either you have to just build coda native, or all those
other systems have to emulate Linux down to the kernel/venus level.  I
could mamake the same claim - why doesn't Linux run NetBSD programs?

When you say 'maintain', do you just mean 'compile binaries from
source'?  This really does not seem hard.  Check out pkgsrc, which
supports all those systems, and has 6.9.1 available.  It makes updating
the software quite easy.

> Thanks for the discussion, I guess a fix is impossible without
> approval and help from the core developers and Jan has a lot of more
> important stuff to take care of.

If you have the resources to do the work, I don't see what would stop
fixing some of these issues, unless the disruption to most outweighs the
gain in cross-ABI functioning.

But IMHO if there are that kind of resources to go into improving coda,
it really needs other things far more badly:

  remove limit on size of directory

  remove requirement that all directories fit in RVM, so that servers
  can simply scale with disk