Well 850,000 files is definitely too many files for our implementation. 
I just ran some quick numbers against our primary data store and it
contains just over 24,000 files and is about 175 GB.  Growing at a rate
of a minimum of 10 GB a month now.

There are four types of files
1. 3 - 10 MB
2. 60 - 120 MB
3. 350 - 600 MB
4. 4-5 GB

Our system currently contain a larger number (80%) of file 1 types. 
Average file size of ~7 MB.

In order to optimize the venus cache for our file allocation, would you
say having the venus cache stay at ~20-30 GB then set the
cachefiles=10000?

Considering our file sizes and settings, the maximum of 10,000 file
with a minimum 3 meg file would spill over 30 GB, so that should be
alright?

Once I change the venus.conf file, what do I need to run in order to
re-initialize the rvm data?


Regards,






>>> Jan Harkes <jaharkes_at_cs.cmu.edu> 10/24/2006 1:56 PM >>>
> I read through some of the mail listings and found a couple of posts
> that states the maximum rvm data file can not be greater then 1GB. 
> Since these posts are from 2003-2004, I am hoping that things have
> changed?

Not really, we are limited by the amount of addressable memory for a
single process. There is about 4GB available on a 32-bit architecture,
but large parts of that address space are already used up by the
kernel
(typically top 1-2 GB of the address space), the executable code of
the
process and shared libraries, the heap and stack, etc. So you might be
able to squeeze 2GB in there, but then upgrading becomes a russian
roulette as some random shared library or DLL may suddenly overlap
with
the RVM memory segment.

> On a Windows XP client, when setting a 20 GB cache size and I trying
to
> start venus I receive the following error message in

However, what you are looking for is using 20GB of diskspace, where
our
RVM limitations are mostly related to the metadata (number of files).
Venus uses a magic number to get from the cache-size to the number of
files, we simply assume the average file is 24KB. So your client is
trying to preallocate metadata for about 850,000 files / 5,000,000 CML
entries etc.

But you can override the number of cachefiles to a more reasonable
value in venus.conf.

    cacheblocks=20971520
    cachefiles=100000	  # assume average file size ~200KB

It may still trigger some problems, I tend to run my clients with
caches
that hold approximately ~8000 - 10000 files.

Jan