radiance + openmosix = :)

Hi all,

I had some time to do a few experiments with
openmosix and radiance during the past weeks.
And, luckily, I've been successful, or at least
now I know a little bit more about both systems ...

Maybe somebody can be interested ...

Probably many of you already know what openmosix is
(and of course _all_ of you should know about radiance,
even if day by day we discover something new about it ...):
as far as I've understood, openmosix is a linux kernel
extension that provides single image clustering
capabilities to a network of linux boxes.
In other words, instead of having a machine with a single
processor, it is possible to see as many processors
as all your networked linux boxes have.

The main advantages of the openmosix approach to do parallel
computation with radiance are:

1. no need to use nfs (it can really be painful to choose
   the right version and to set up the lockd daemon with
   linux)
2. no need to login to different machines to launch the rpiece
   processes (no rsh or ssh or telnet or whatever needed)
3. it works with rholo -n x, when x>1

What you need: a few computers laying around, a linux distro
(I prefer and suggest debian or knoppix/clusterknppix, maybe
gentoo can also be practical), a newtork switch + cables,
time, patience.

At work, I've been able to put over and under my desk (kidnapped)
1, 2, 3, ..., 8 pc's, some old, some new (from 433Mhz PIII to 2.2Ghz PIV),
and I started doing some tests in my very little spare time.
It is a little bit hot now, and I get a headache after 5 minutes,
but ...

First I installed gnu/debian on each of them (well, I installed
it on one and then cloned it with rsync), and then recompiled
the linux kernel (2.4.26) with the unofficial openmosix patch
from http://openmosix.snarc.org/download: I used the unofficial
one because it is more updated, and I needed to have at least
kernel 2.4.26 for a couple of hardware devices on my laptop to work:
it is very important for all the machines in the network to
have the same kernel and openmosix versions, otherwise they
won't communicate. There are no openmosix patches for 2.6.x kernels,
yet.

After installing the kernel and compiling and installing radiance
(I used the optimisations for the slowest processors in the cluster,
PIII), I verified that everything was working: the new openmosix releases
include an autodiscovery daemon that takes care of automaticaly
adding new nodes to the cluster when they are available, and also use a
special load balancing algorithm to automatically migrate the processes to
the faster nodes.
Debian has two openmosix packages, one for the patch (the patch is not very
updated, 2.4.24, but also installs the daemon init scripts) and another for
the userspace programs, to monitor the cluster and manage the migration
process. There also is an additional graphical interface, openmosixview,
quite helpful.

The first test I did was with rholo, and it worked pretty well with the
network of slowest computers: when adding the faster ones, some processes
were migrating to the fastest machines, other were spending a lot of time
in deciding what to do, adding overhead to the system and slowing down
the simulation. With rpiece it was even worse, and I was starting losing
my faith ...

As you may have noticed, my system is very asymmetrical, and this was
the cause of many failures.

After that, I've tried with the fastest ones (2.0 and 2.2GHz PIV), and
after some failures I realized that I had to launch the rpiece processes
in a different way, using some of the openmosix userspace tools.
Now I use "runhome oconv [options]" to compile the octree on the fastest
machine, and I launch a first rpiece process that stays on the first
machine ("runhome rpiece @args &") and several new rpiece processes that
are able to migrate to the other fast nodes with "mosrun -t 1 -d 700 rpiece
@args &". I understood that it is better to turn off the slower pc's or to
keep them in a different network segment, otherwise they will make the
system slower and slower. So a symmetrical cluster is far better.

To make it possible to compare my results, I've benchmarked my 2-node cluster
with the bench2 package as taken from Mark Stock's wonderful site, and here
are the results (radiance was compiled with PIV optimisations).

standard bench2 on the 2.0GHz PIV
21.92user 0.85system 0:23.45elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60294major+31417minor)pagefaults 0swaps
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.004r hours on leviathan
rpict: 2740602 rays, 32.40% after 0.012u 0.000s 0.013r hours on leviathan
rpict: 5594096 rays, 42.60% after 0.020u 0.000s 0.021r hours on leviathan
rpict: 8454012 rays, 48.60% after 0.028u 0.000s 0.029r hours on leviathan
rpict: 11120053 rays, 52.20% after 0.036u 0.000s 0.038r hours on leviathan
rpict: 13856057 rays, 56.10% after 0.045u 0.000s 0.047r hours on leviathan
rpict: 16533987 rays, 60.60% after 0.054u 0.001s 0.055r hours on leviathan
rpict: 19224383 rays, 66.00% after 0.062u 0.001s 0.064r hours on leviathan
rpict: 21936654 rays, 71.10% after 0.071u 0.001s 0.072r hours on leviathan
rpict: 24532107 rays, 75.90% after 0.079u 0.001s 0.081r hours on leviathan
rpict: 27207234 rays, 81.30% after 0.087u 0.001s 0.089r hours on leviathan
rpict: 29704006 rays, 100.00% after 0.095u 0.001s 0.097r hours on leviathan
341.47user 2.70system 5:49.37elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60349major+31607minor)pagefaults 0swaps
0.46user 0.00system 0:00.52elapsed 87%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (484major+66minor)pagefaults 0swaps

So the rpict process has ended after 5:49.

By changing the doit script to this:

···

------------------------------------------------------------------------------
rm -f scene.oct scene.pic scene.amb syncfile args
/usr/bin/time runhome oconv scene.rad > scene.oct
echo 4 4 > syncfile
echo "-F syncfile -t 30 -vf v1.vf -af scene.amb -ab 1 -x 1000 -y 1000 \
-aa 0.2 -ar 32 -ad 128 -as 0 -lr 6 -lw 0.005 -o scene.pic scene.oct" > args
/usr/bin/time runhome rpiece @args &
/usr/bin/time mosrun -t 1 -d 700 rpiece @args &
------------------------------------------------------------------------------

I got the following result:

21.85user 0.91system 0:22.94elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60294major+31417minor)pagefaults 0swaps
FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 1.5
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan
rpict: 3987 rays, 100.00% after 0.004u 0.000s 0.008r hours on leviathan
FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -0.5
rpict: 3987 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan
FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 0.5
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan
rpict: 4154 rays, 100.00% after 0.004u 0.000s 0.009r hours on leviathan
FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -1.5
rpict: 4154 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan
rpict: 726110 rays, 100.00% after 0.006u 0.000s 0.012r hours on leviathan
FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 1.5
rpict: 726110 rays, 0.00% after 0.006u 0.000s 0.012r hours on leviathan
rpict: 492809 rays, 100.00% after 0.005u 0.001s 0.011r hours on leviathan
FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 0.5
rpict: 492809 rays, 0.00% after 0.005u 0.001s 0.011r hours on leviathan
rpict: 1324193 rays, 100.00% after 0.008u 0.001s 0.014r hours on leviathan
FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -0.5
rpict: 1324193 rays, 0.00% after 0.008u 0.001s 0.014r hours on leviathan
rpict: 3678661 rays, 88.80% after 0.013u 0.001s 0.019r hours on leviathan
rpict: 3902848 rays, 100.00% after 0.014u 0.001s 0.020r hours on leviathan
FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -1.5
rpict: 3902848 rays, 0.00% after 0.014u 0.001s 0.020r hours on leviathan
rpict: 3959908 rays, 39.60% after 0.016u 0.001s 0.022r hours on leviathan
rpict: 5912685 rays, 100.00% after 0.022u 0.001s 0.029r hours on leviathan
FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 1.5
rpict: 5912685 rays, 0.00% after 0.022u 0.001s 0.029r hours on leviathan
rpict: 6612613 rays, 100.00% after 0.021u 0.001s 0.028r hours on leviathan
FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 0.5
rpict: 6612613 rays, 0.00% after 0.021u 0.001s 0.028r hours on leviathan
rpict: 6599351 rays, 100.00% after 0.024u 0.001s 0.031r hours on leviathan
FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -0.5
rpict: 6599351 rays, 0.00% after 0.024u 0.001s 0.031r hours on leviathan
rpict: 8845669 rays, 69.60% after 0.027u 0.001s 0.037r hours on leviathan
rpict: 8907044 rays, 18.00% after 0.031u 0.001s 0.039r hours on leviathan
rpict: 10518921 rays, 100.00% after 0.031u 0.001s 0.044r hours on leviathan
FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -1.5
rpict: 10518921 rays, 0.00% after 0.031u 0.001s 0.045r hours on leviathan
rpict: 11029597 rays, 40.80% after 0.038u 0.001s 0.048r hours on leviathan
rpict: 13086055 rays, 51.60% after 0.038u 0.001s 0.053r hours on leviathan
rpict: 13229033 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan
FRAME 7: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 1.5
rpict: 13229033 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan
rpict: 13233114 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan
FRAME 8: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 0.5
rpict: 13233114 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan
rpict: 13263283 rays, 100.00% after 0.039u 0.001s 0.054r hours on leviathan
FRAME 9: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -0.5
rpict: 13263283 rays, 0.00% after 0.039u 0.001s 0.054r hours on leviathan
rpict: 13306793 rays, 75.60% after 0.046u 0.001s 0.056r hours on leviathan
rpict: 14642671 rays, 100.00% after 0.042u 0.001s 0.057r hours on leviathan
FRAME 10: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -1.5
rpict: 14642671 rays, 0.00% after 0.042u 0.001s 0.057r hours on leviathan
rpict: 14338103 rays, 100.00% after 0.049u 0.001s 0.059r hours on leviathan
177.45user 3.49system 3:33.27elapsed 84%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60493major+32009minor)pagefaults 0swaps
rpict: 15387552 rays, 100.00% after 0.044u 0.001s 0.059r hours on leviathan
159.67user 2.77system 3:40.42elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60894major+33670minor)pagefaults 0swaps

So the rpiece processes ended after 3:40 (~63% of the previous time)
-> very promising, isn't it?

You may have noticed that I've also shared the ambient file between the
two processes (I haven't cheated, it was erased at the beginning of
the simulation), and since the file was accessed on the same computer
there was no need to set up a network file system, and there was no need to
think about file permissions and so on ...

Hope you may find this useful ...

Sorry for not writing very frequently.
And for writing this much this time (!).

I hope to see you all at the 2004 workshop
(any news?).

--
Francesco Anselmo
[email protected]

Hi Francesco,

I can partly confirm the observations from your parallel rendering experiments. I didn't use openmosix so far, but some time ago I wrote a parallel Radiance using the PVM-Library ('Parallel Virtual Machine').

This PVM provides a collection of routines for interprocess communication on different sorts of machines. Instead of using rpiece I changed the Radiance code to produce a set of master and slave modules, one for the distribution of the work, the others for doing the rendering and collecting the results to produce the final picture. By this, I also could avoid the Linux lock manager problem for the ambient files, as I shared ambient values among the processes with help of the PVM routines.

Although PVM offers to connect all sorts of machines and OS-es together, it is of course advisable to use a symmetric cluster built from machines with comparable speed. (BTW, by this you also can use the same data format on all machines and don't need to codec data into a separate exchange buffer).

I started with 4 equal 400MHz processors, when I later added faster machines, I quickly observed that, e.g the two 1.2 GHz machines already finished almost all the blocks of the divided image when the four old ones hardly managed to trace one block each. So it really doesn't pay off to add slow machines to your cluster.

Additionally, the overhead from parallelization was considerable. I didn't really care about benchmarking, but I remember to have had an overhead of rougly about 20% (with my PVM version), meaning when using e.g. 4 machines, rendering was not 4 times but only ~3.2 times faster.( -nb: PVM does no migrating, at least not the version from several years ago). Its important to make the blocks small, because you almost always will end up with the following: all machines finished long time ago but only one still working on it's last block :slight_smile:

Another totally different problem I observed was that sometimes the block boundaries have been visible, i.e. the image showed a checkered pattern, different blocks appeared with different brightnesses. I came so far to attribute this effect to the direct calculation, but haven't completely found out where exactly the cause lies.

Nevertheless that migrating of openmosix sounds interesting, if I find the time I'd like to have a look at it..

so long & see you in Fribourg

-Carsten

Hi Carsten,

Using the PVM library should be quite difficult compared to
using openmosix: I forgot to mention that probably the best
advantage of openmosix is that in many cases there is no need to change
the application source code.

But openmosix doesn't work with heterogeneous clusters, being a
linux based product ...

Thank you for your feedback, and for the great job you've done
on Radzilla: unfortunately I only had the time to read the
documentation, but I hope to start using it very soon ...

See you soon.

Francesco

Hi,

one small note, did you report your experiences to the openmosix team? They have a lot of docs on their website, and a howto, which mentions e.g. povray with openmosix. A hint that you have been able to use with radiance (the text of your mail is quite a help) might be interesting on the openmosix-website?

CU Lars.

···

--
Lars O. Grobe
[email protected]

Hi Lars,

one small note, did you report your experiences to the openmosix team?

Not yet, and I can't afford subscribing to another mailing-list :wink: !!!
I did a quick search in the openmosix mailing-lists and there doesn't seem
to be any trace of Radiance in their discussions ...

A hint that you have been able to use with
radiance (the text of your mail is quite a help) might be interesting
on the openmosix-website?

Maybe, if I manage to write a short howto, I will send it to the om team.
Thank you!

···

--
Francesco