Hi all,

I had some time to do a few experiments with

openmosix and radiance during the past weeks.

And, luckily, I've been successful, or at least

now I know a little bit more about both systems ...

Maybe somebody can be interested ...

Probably many of you already know what openmosix is

(and of course _all_ of you should know about radiance,

even if day by day we discover something new about it ...):

as far as I've understood, openmosix is a linux kernel

extension that provides single image clustering

capabilities to a network of linux boxes.

In other words, instead of having a machine with a single

processor, it is possible to see as many processors

as all your networked linux boxes have.

The main advantages of the openmosix approach to do parallel

computation with radiance are:

1. no need to use nfs (it can really be painful to choose

the right version and to set up the lockd daemon with

linux)

2. no need to login to different machines to launch the rpiece

processes (no rsh or ssh or telnet or whatever needed)

3. it works with rholo -n x, when x>1

What you need: a few computers laying around, a linux distro

(I prefer and suggest debian or knoppix/clusterknppix, maybe

gentoo can also be practical), a newtork switch + cables,

time, patience.

At work, I've been able to put over and under my desk (kidnapped)

1, 2, 3, ..., 8 pc's, some old, some new (from 433Mhz PIII to 2.2Ghz PIV),

and I started doing some tests in my very little spare time.

It is a little bit hot now, and I get a headache after 5 minutes,

but ...

First I installed gnu/debian on each of them (well, I installed

it on one and then cloned it with rsync), and then recompiled

the linux kernel (2.4.26) with the unofficial openmosix patch

from http://openmosix.snarc.org/download: I used the unofficial

one because it is more updated, and I needed to have at least

kernel 2.4.26 for a couple of hardware devices on my laptop to work:

it is very important for all the machines in the network to

have the same kernel and openmosix versions, otherwise they

won't communicate. There are no openmosix patches for 2.6.x kernels,

yet.

After installing the kernel and compiling and installing radiance

(I used the optimisations for the slowest processors in the cluster,

PIII), I verified that everything was working: the new openmosix releases

include an autodiscovery daemon that takes care of automaticaly

adding new nodes to the cluster when they are available, and also use a

special load balancing algorithm to automatically migrate the processes to

the faster nodes.

Debian has two openmosix packages, one for the patch (the patch is not very

updated, 2.4.24, but also installs the daemon init scripts) and another for

the userspace programs, to monitor the cluster and manage the migration

process. There also is an additional graphical interface, openmosixview,

quite helpful.

The first test I did was with rholo, and it worked pretty well with the

network of slowest computers: when adding the faster ones, some processes

were migrating to the fastest machines, other were spending a lot of time

in deciding what to do, adding overhead to the system and slowing down

the simulation. With rpiece it was even worse, and I was starting losing

my faith ...

As you may have noticed, my system is very asymmetrical, and this was

the cause of many failures.

After that, I've tried with the fastest ones (2.0 and 2.2GHz PIV), and

after some failures I realized that I had to launch the rpiece processes

in a different way, using some of the openmosix userspace tools.

Now I use "runhome oconv [options]" to compile the octree on the fastest

machine, and I launch a first rpiece process that stays on the first

machine ("runhome rpiece @args &") and several new rpiece processes that

are able to migrate to the other fast nodes with "mosrun -t 1 -d 700 rpiece

@args &". I understood that it is better to turn off the slower pc's or to

keep them in a different network segment, otherwise they will make the

system slower and slower. So a symmetrical cluster is far better.

To make it possible to compare my results, I've benchmarked my 2-node cluster

with the bench2 package as taken from Mark Stock's wonderful site, and here

are the results (radiance was compiled with PIV optimisations).

standard bench2 on the 2.0GHz PIV

21.92user 0.85system 0:23.45elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (60294major+31417minor)pagefaults 0swaps

rpict: 0 rays, 0.00% after 0.004u 0.000s 0.004r hours on leviathan

rpict: 2740602 rays, 32.40% after 0.012u 0.000s 0.013r hours on leviathan

rpict: 5594096 rays, 42.60% after 0.020u 0.000s 0.021r hours on leviathan

rpict: 8454012 rays, 48.60% after 0.028u 0.000s 0.029r hours on leviathan

rpict: 11120053 rays, 52.20% after 0.036u 0.000s 0.038r hours on leviathan

rpict: 13856057 rays, 56.10% after 0.045u 0.000s 0.047r hours on leviathan

rpict: 16533987 rays, 60.60% after 0.054u 0.001s 0.055r hours on leviathan

rpict: 19224383 rays, 66.00% after 0.062u 0.001s 0.064r hours on leviathan

rpict: 21936654 rays, 71.10% after 0.071u 0.001s 0.072r hours on leviathan

rpict: 24532107 rays, 75.90% after 0.079u 0.001s 0.081r hours on leviathan

rpict: 27207234 rays, 81.30% after 0.087u 0.001s 0.089r hours on leviathan

rpict: 29704006 rays, 100.00% after 0.095u 0.001s 0.097r hours on leviathan

341.47user 2.70system 5:49.37elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (60349major+31607minor)pagefaults 0swaps

0.46user 0.00system 0:00.52elapsed 87%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (484major+66minor)pagefaults 0swaps

So the rpict process has ended after 5:49.

By changing the doit script to this:

## ···

------------------------------------------------------------------------------

rm -f scene.oct scene.pic scene.amb syncfile args

/usr/bin/time runhome oconv scene.rad > scene.oct

echo 4 4 > syncfile

echo "-F syncfile -t 30 -vf v1.vf -af scene.amb -ab 1 -x 1000 -y 1000 \

-aa 0.2 -ar 32 -ad 128 -as 0 -lr 6 -lw 0.005 -o scene.pic scene.oct" > args

/usr/bin/time runhome rpiece @args &

/usr/bin/time mosrun -t 1 -d 700 rpiece @args &

------------------------------------------------------------------------------

I got the following result:

21.85user 0.91system 0:22.94elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (60294major+31417minor)pagefaults 0swaps

FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 1.5

rpict: 0 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan

rpict: 3987 rays, 100.00% after 0.004u 0.000s 0.008r hours on leviathan

FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -0.5

rpict: 3987 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan

FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 0.5

rpict: 0 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan

rpict: 4154 rays, 100.00% after 0.004u 0.000s 0.009r hours on leviathan

FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -1.5

rpict: 4154 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan

rpict: 726110 rays, 100.00% after 0.006u 0.000s 0.012r hours on leviathan

FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 1.5

rpict: 726110 rays, 0.00% after 0.006u 0.000s 0.012r hours on leviathan

rpict: 492809 rays, 100.00% after 0.005u 0.001s 0.011r hours on leviathan

FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 0.5

rpict: 492809 rays, 0.00% after 0.005u 0.001s 0.011r hours on leviathan

rpict: 1324193 rays, 100.00% after 0.008u 0.001s 0.014r hours on leviathan

FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -0.5

rpict: 1324193 rays, 0.00% after 0.008u 0.001s 0.014r hours on leviathan

rpict: 3678661 rays, 88.80% after 0.013u 0.001s 0.019r hours on leviathan

rpict: 3902848 rays, 100.00% after 0.014u 0.001s 0.020r hours on leviathan

FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -1.5

rpict: 3902848 rays, 0.00% after 0.014u 0.001s 0.020r hours on leviathan

rpict: 3959908 rays, 39.60% after 0.016u 0.001s 0.022r hours on leviathan

rpict: 5912685 rays, 100.00% after 0.022u 0.001s 0.029r hours on leviathan

FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 1.5

rpict: 5912685 rays, 0.00% after 0.022u 0.001s 0.029r hours on leviathan

rpict: 6612613 rays, 100.00% after 0.021u 0.001s 0.028r hours on leviathan

FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 0.5

rpict: 6612613 rays, 0.00% after 0.021u 0.001s 0.028r hours on leviathan

rpict: 6599351 rays, 100.00% after 0.024u 0.001s 0.031r hours on leviathan

FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -0.5

rpict: 6599351 rays, 0.00% after 0.024u 0.001s 0.031r hours on leviathan

rpict: 8845669 rays, 69.60% after 0.027u 0.001s 0.037r hours on leviathan

rpict: 8907044 rays, 18.00% after 0.031u 0.001s 0.039r hours on leviathan

rpict: 10518921 rays, 100.00% after 0.031u 0.001s 0.044r hours on leviathan

FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -1.5

rpict: 10518921 rays, 0.00% after 0.031u 0.001s 0.045r hours on leviathan

rpict: 11029597 rays, 40.80% after 0.038u 0.001s 0.048r hours on leviathan

rpict: 13086055 rays, 51.60% after 0.038u 0.001s 0.053r hours on leviathan

rpict: 13229033 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan

FRAME 7: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 1.5

rpict: 13229033 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan

rpict: 13233114 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan

FRAME 8: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 0.5

rpict: 13233114 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan

rpict: 13263283 rays, 100.00% after 0.039u 0.001s 0.054r hours on leviathan

FRAME 9: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -0.5

rpict: 13263283 rays, 0.00% after 0.039u 0.001s 0.054r hours on leviathan

rpict: 13306793 rays, 75.60% after 0.046u 0.001s 0.056r hours on leviathan

rpict: 14642671 rays, 100.00% after 0.042u 0.001s 0.057r hours on leviathan

FRAME 10: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882

-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -1.5

rpict: 14642671 rays, 0.00% after 0.042u 0.001s 0.057r hours on leviathan

rpict: 14338103 rays, 100.00% after 0.049u 0.001s 0.059r hours on leviathan

177.45user 3.49system 3:33.27elapsed 84%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (60493major+32009minor)pagefaults 0swaps

rpict: 15387552 rays, 100.00% after 0.044u 0.001s 0.059r hours on leviathan

159.67user 2.77system 3:40.42elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (60894major+33670minor)pagefaults 0swaps

So the rpiece processes ended after 3:40 (~63% of the previous time)

-> very promising, isn't it?

You may have noticed that I've also shared the ambient file between the

two processes (I haven't cheated, it was erased at the beginning of

the simulation), and since the file was accessed on the same computer

there was no need to set up a network file system, and there was no need to

think about file permissions and so on ...

Hope you may find this useful ...

Sorry for not writing very frequently.

And for writing this much this time (!).

I hope to see you all at the 2004 workshop

(any news?).

--

Francesco Anselmo

[email protected]