rpiece question

atelier_iebele_abel1 · July 23, 2008, 9:35pm

Hi all,

Does any of you wants to share an example of rpiece, using multiple (say 4) processes on a single machine, sharing 1 ambient file, resulting in 1 single image?

The manpage gives this example:

goober% echo 1 8 > syncfile
goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args
goo1ber% rpiece @args &

How do I let rpiece run 4 processes simultaneous? Just run 4 processes in the background?
How does an ambient file works using rpiece? Do I just need one, or as many as there are "pieces"?
Where are temporary files stored, so I can check their progress?

-Iebele

Jack_de_Valpine · July 23, 2008, 10:07pm

Hi Iebele,

A few brief answers to start in line below....

-Jack

iebele wrote:

Hi all,
Does any of you wants to share an example of rpiece, using multiple (say 4) processes on a single machine, sharing 1 ambient file, resulting in 1 single image?
The manpage gives this example:
goober% echo 1 8 > syncfile
goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args
goo1ber% rpiece @args &
How do I let rpiece run 4 processes simultaneous? Just run 4 processes in the background?

yes simply repeat the command "rpiece @args&" for however many processes you want

How does an ambient file works using rpiece? Do I just need one, or as many as there are "pieces"?

Just one ambient file. This would be included in your args.

Where are temporary files stored, so I can check their progress?

If you include an error file output in your args then you can check this as one diagnostic. Additionally you can check the contents of "syncfile" to get a sense of how many "pieces" have been completed. Lastly you can do a pfilt on the unfinished image and view the results to visually see things. Files will be stored as you desire, so it is up to you to organize things the way you want.

I would suggest perhaps doing a smaller image or lower rendering parameters so you can see how things work before diving into something more involved.

···

-Iebele
------------------------------------------------------------------------

_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction

atelier_iebele_abel1 · July 23, 2008, 10:17pm

Thanks Jack,
A brief question in line in return...

Lastly you can do a pfilt on the unfinished image and view the results to visually see things. Files will be stored as you desire, so it is up to you to organize things the way you want.

atelier_iebele_abel1 · July 23, 2008, 10:20pm

Thanks Jack,
A brief question in line in return...

I underestand that the -o option sets the final image.
But where can I find the temporary files , i.e. the "pieces" ?

-Iebele

···

From: iebele
  To: Radiance general discussion
  Sent: Thursday, July 24, 2008 12:17 AM
  Subject: Re: [Radiance-general] rpiece question

Thanks Jack,
A brief question in line in return...

Lastly you can do a pfilt on the unfinished image and view the results to visually see things. Files will be stored as you desire, so it is up to you to organize things the way you want.

Mark_Stock · July 23, 2008, 11:41pm

Iebele,

The runsmp script contained in my Radiance benchmark tests demonstrates this. It's what I use to render my large art images.

http://mark.technolope.org/pages/rad_bench.html
http://mark.technolope.org/radmisc/bench4.tar.gz

Mark

PS. here's the script rewritten for 4 CPUs:

···

-------------
#!/bin/sh

[ -z "$NCPU" ] && NCPU=4
[ -z "$COLS" ] && COLS=1
[ -z "$ROWS" ] && ROWS=64

echo $COLS $ROWS > syncfile
rm -f pp

while [ "$NCPU" -gt 0 ]
do
         echo "running proc $NCPU"
         rpiece -v -PP pp -F syncfile @viewpoint @options @args &
         NCPU=`expr $NCPU - 1`
done
wait

rm -f syncfile pp

exit 0
-------------

On Wed, 23 Jul 2008, iebele wrote:

Hi all,

Does any of you wants to share an example of rpiece, using multiple (say 4) processes on a single machine, sharing 1 ambient file, resulting in 1 single image?

The manpage gives this example:

goober% echo 1 8 > syncfile
goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args
goo1ber% rpiece @args &

How do I let rpiece run 4 processes simultaneous? Just run 4 processes in the background?
How does an ambient file works using rpiece? Do I just need one, or as many as there are "pieces"?
Where are temporary files stored, so I can check their progress?

-Iebele

Ilya_A_Zimnovich · July 24, 2008, 5:37am

Hello,

You can try our scripts which do exactly what you want:
http://www.bakharev.org/index.php?option=com_content&task=category&sectionid=5&id=23&Itemid=38&lang=english
In fact they're wrappers to rpict and rtrace and take the same options.

Best Regards,
Ilya

iebele wrote:

···

Hi all,

Does any of you wants to share an example of rpiece, using multiple (say
4) processes on a single machine, sharing 1 ambient file, resulting in 1
single image?

The manpage gives this example:

goober% echo 1 8 > syncfile
goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args
goo1ber% rpiece @args &

How do I let rpiece run 4 processes simultaneous? Just run 4 processes
in the background?
How does an ambient file works using rpiece? Do I just need one, or as
many as there are "pieces"?
Where are temporary files stored, so I can check their progress?

-Iebele

------------------------------------------------------------------------

_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

Jack_de_Valpine · July 24, 2008, 1:22pm

Hi Iebele,

All the "pieces" get written out into the file specified by the -o option. So there are no individual image pieces if that makes sense.

To avoid some issues relating to speed of file locking for rpiece jobs shared over multiple compute nodes (such as in a cluster with a shared nfs server), I have actually hacked rpiece so that it will in fact write out the individual pieces as image files. These can then be assembled into the final image after the fact. I am willing to make this available if there is interest.

Regards,

-Jack de Valpine

iebele wrote:

···

Thanks Jack,
A brief question in line in return...
I underestand that the -o option sets the final image.
But where can I find the temporary files , i.e. the "pieces" ?
-Iebele

    *From:* iebele <mailto:info@iebele.nl>
    *To:* Radiance general discussion
    <mailto:radiance-general@radiance-online.org>
    *Sent:* Thursday, July 24, 2008 12:17 AM
    *Subject:* Re: [Radiance-general] rpiece question

    Thanks Jack,
    A brief question in line in return...

                  Lastly you can do a pfilt on the unfinished image and view
        the results to visually see things. Files will be stored as
        you desire, so it is up to you to organize things the way you
        want.

------------------------------------------------------------------------

_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction

atelier_iebele_abel1 · July 24, 2008, 2:01pm

Hi Mark, Ilya,

Thanks for your help, rpiece runs cool now, the script is very helpfull in understanding the common usage of rpiece,

Iebele

···

----- Original Message ----- From: "Mark Stock" <mstock@umich.edu>
To: "Radiance general discussion" <radiance-general@radiance-online.org>
Sent: Thursday, July 24, 2008 1:41 AM
Subject: Re: [Radiance-general] rpiece question

Iebele,

The runsmp script contained in my Radiance benchmark tests demonstrates this. It's what I use to render my large art images.

http://mark.technolope.org/pages/rad_bench.html
http://mark.technolope.org/radmisc/bench4.tar.gz

Mark

PS. here's the script rewritten for 4 CPUs:
-------------
#!/bin/sh

[ -z "$NCPU" ] && NCPU=4
[ -z "$COLS" ] && COLS=1
[ -z "$ROWS" ] && ROWS=64

echo $COLS $ROWS > syncfile
rm -f pp

while [ "$NCPU" -gt 0 ]
do
        echo "running proc $NCPU"
        rpiece -v -PP pp -F syncfile @viewpoint @options @args &
        NCPU=`expr $NCPU - 1`
done
wait

rm -f syncfile pp

exit 0
-------------

On Wed, 23 Jul 2008, iebele wrote:

Hi all,

Does any of you wants to share an example of rpiece, using multiple (say 4) processes on a single machine, sharing 1 ambient file, resulting in 1 single image?

The manpage gives this example:

goober% echo 1 8 > syncfile
goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args
goo1ber% rpiece @args &

How do I let rpiece run 4 processes simultaneous? Just run 4 processes in the background?
How does an ambient file works using rpiece? Do I just need one, or as many as there are "pieces"?
Where are temporary files stored, so I can check their progress?

-Iebele

_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

atelier_iebele_abel1 · July 24, 2008, 2:04pm

Hi Jack,

Yes, it would be helpfull to have acces to the individual pieces.
If you want to share your hack, I would be very pleased with that.

-Iebele

···

----- Original Message -----
  From: Jack de Valpine
  To: Radiance general discussion
  Cc: iebele
  Sent: Thursday, July 24, 2008 3:22 PM
  Subject: Re: [Radiance-general] rpiece question

Hi Iebele,

All the "pieces" get written out into the file specified by the -o option. So there are no individual image pieces if that makes sense.

To avoid some issues relating to speed of file locking for rpiece jobs shared over multiple compute nodes (such as in a cluster with a shared nfs server), I have actually hacked rpiece so that it will in fact write out the individual pieces as image files. These can then be assembled into the final image after the fact. I am willing to make this available if there is interest.

Regards,

-Jack de Valpine

  iebele wrote:
    Thanks Jack,
    A brief question in line in return...

I underestand that the -o option sets the final image.
But where can I find the temporary files , i.e. the "pieces" ?

    -Iebele
      From: iebele
      To: Radiance general discussion
      Sent: Thursday, July 24, 2008 12:17 AM
      Subject: Re: [Radiance-general] rpiece question

Thanks Jack,
A brief question in line in return...

Lastly you can do a pfilt on the unfinished image and view the results to visually see things. Files will be stored as you desire, so it is up to you to organize things the way you want.
----------------------------------------------------------------------------
_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction

------------------------------------------------------------------------------

  _______________________________________________
  Radiance-general mailing list
  Radiance-general@radiance-online.org
  http://www.radiance-online.org/mailman/listinfo/radiance-general

Rob_Guglielmetti2 · July 24, 2008, 3:17pm

Jack de Valpine wrote:

To avoid some issues relating to speed of file locking for rpiece jobs shared over multiple compute nodes (such as in a cluster with a shared nfs server), I have actually hacked rpiece so that it will in fact write out the individual pieces as image files. These can then be assembled into the final image after the fact. I am willing to make this available if there is interest.

There is interest... =8-)

On a related note, does anyone here have experience with Oracle's cluster file system (ocfs)? My IT guy here thinks this filesystem might eliminate all the nasty file locking problems that have plagued rpiece on linux for years. I was about to test this out myself when I get some free time, as we now have two nodes that are running it, but I haven't had a chance yet.

- Rob

Jack_de_Valpine · July 24, 2008, 5:59pm

Hi Rob,

I will package up a few things and make sure that it compiles on 3.9.

Just out of curiosity what locking issues have been plaguing you? For rpiece there are essentially 3 files that are locked/unlocked:

   1. syncfile - each process checks this to see what it can render next
   2. output image - each process needs to lock and write out its piece
      to the the output file
   3. ambient file - this is actually managed outside of rpiece in how
      rpict works I believe.

-Jack

Rob Guglielmetti wrote:

···

Jack de Valpine wrote:

To avoid some issues relating to speed of file locking for rpiece jobs shared over multiple compute nodes (such as in a cluster with a shared nfs server), I have actually hacked rpiece so that it will in fact write out the individual pieces as image files. These can then be assembled into the final image after the fact. I am willing to make this available if there is interest.

There is interest... =8-)
On a related note, does anyone here have experience with Oracle's cluster file system (ocfs)? My IT guy here thinks this filesystem might eliminate all the nasty file locking problems that have plagued rpiece on linux for years. I was about to test this out myself when I get some free time, as we now have two nodes that are running it, but I haven't had a chance yet.
- Rob

_______________________________________________
Radiance-general mailing list
Radiance-general@radiance-online.org
http://www.radiance-online.org/mailman/listinfo/radiance-general

--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction