SMP file busy problem on iterative rendering in Linux

hdri
1

Dear Radiance Users,

I've assembled a simple script that iterates through a folder containing octrees, and renders using SMP. On Mac, everything works as expected. However, on Linux, I get errors suggesting both the syncfile and ppfile are still locked by the OS. I've attempted two solutions, including adding 'sleep 5' after pfilt (first attempt, shown below), and then by externalizing the while loop, with an ending 'exit 0' (second attempt, not shown here), but neither approach fixed the problem.

Here is one version of code which works on Mac but not in Linux:

for file in obj/*.oct
do
     filename="${file%.*}"
     rm -f syncfile argsfile ppfile
     echo 1 64 > syncfile
     cat syncfile
     echo -F syncfile @arguments -vf room.vf -o $filename.unf $filename.oct > argsfile
     cat argsfile
     NCPU=2
     while [ "$NCPU" -gt 0 ]
     do
         echo "running proc $NCPU"
         rpiece -v -PP ppfile @argsfile &
         sleep 5
         NCPU=`expr $NCPU - 1`
     done
     wait
     echo "Filtering File - $filename.unf"
     pfilt -1 -e -2 -m .25 -r .6 -x /3 -y /3 $filename.unf > $filename.hdr
     sleep 5
     rm -f syncfile argsfile ppfile
done

Interestingly, I do not get an error saying "cannot remove 'argsfile' ". After the first image is successfully rendered, I get the following errors at each successive iteration, indicating the rm only works once (line 4) in the first iteration:

rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy
rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy

Since the error isn't fatal, the machine keeps rendering, and ends up rendering the same (first) image over and over. Has anyone run into this problem and found a solution? Any advice would be greatly appreciated.

Thanks,
Chris

2

Thanks Greg and Iebele,

Seems like there are two possible solutions: 1) per Greg's suggestion, monitor and wait for the files to unlock (or kill them); or, 2) create octrees at each iteration (if I'm understanding Iebele's suggestion correctly). My guess is that creating the octrees at the beginning of each iteration gives the OS plenty of time for the unlock to cascade. Is that a reasonable possibility/assumption, or did I misunderstand the second solution?

I will try these (and any other suggestions), and report if successful...

Thanks again!
Chris

···

On 1/31/14 3:43 PM, Gregory J. Ward wrote:

Let's keep this on radiance-general.

Thanks.
-Greg

Begin forwarded message:

*From: *iebele abel <[email protected] <mailto:[email protected]>>
*Date: *January 31, 2014 12:03:14 PM PST
*To: *High Dynamic Range Imaging <[email protected] <mailto:[email protected]>>
*Subject: **RE: [HDRI] SMP file busy problem on iterative rendering in Linux*

Dear Chris,

I had something similar, using a similar script. That situation - a lock on the pp file - was solved after all octrees were compiled again, although rvu and rpict did not complain. Somehow rpiece locked the pp file, however. Strange... Good luck,

iebele

---- Chris Kallie schreef ----

Dear Radiance Users,

I've assembled a simple script that iterates through a folder containing
octrees, and renders using SMP. On Mac, everything works as expected.
However, on Linux, I get errors suggesting both the syncfile and ppfile
are still locked by the OS. I've attempted two solutions, including
adding 'sleep 5' after pfilt (first attempt, shown below), and then by
externalizing the while loop, with an ending 'exit 0' (second attempt,
not shown here), but neither approach fixed the problem.

Here is one version of code which works on Mac but not in Linux:

for file in obj/*.oct
do
   filename="${file%.*}"
   rm -f syncfile argsfile ppfile
   echo 1 64 > syncfile
   cat syncfile
   echo -F syncfile @arguments -vf room.vf -o $filename.unf
$filename.oct > argsfile
   cat argsfile
   NCPU=2
   while [ "$NCPU" -gt 0 ]
   do
       echo "running proc $NCPU"
       rpiece -v -PP ppfile @argsfile &
       sleep 5
       NCPU=`expr $NCPU - 1`
   done
   wait
   echo "Filtering File - $filename.unf"
   pfilt -1 -e -2 -m .25 -r .6 -x /3 -y /3 $filename.unf > $filename.hdr
   sleep 5
   rm -f syncfile argsfile ppfile
done

Interestingly, I do not get an error saying "cannot remove 'argsfile' ".
After the first image is successfully rendered, I get the following
errors at each successive iteration, indicating the rm only works once
(line 4) in the first iteration:

rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy
rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy

Since the error isn't fatal, the machine keeps rendering, and ends up
rendering the same (first) image over and over. Has anyone run into this
problem and found a solution? Any advice would be greatly appreciated.

Thanks,
Chris

_______________________________________________
HDRI mailing list
[email protected] <mailto:[email protected]>
http://www.radiance-online.org/mailman/listinfo/hdri

*From: *[email protected] <mailto:[email protected]>
*Date: *January 31, 2014 12:34:20 PM PST
*Subject: **confirm 20969091e7e2e4da0d2a03094f2222341b41133d*

If you reply to this message, keeping the Subject: header intact,
Mailman will discard the held message. Do this if the message is
spam. If you reply to this message and include an Approved: header
with the list password in it, the message will be approved for posting
to the list. The Approved: header can also appear in the first line
of the body of the reply.

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general

3

Dear renderers,

My solution (to the SMP file busy problem reported earlier) is to not use Radiance on modern virtual machines (in this case: Virtual Box running Ubuntu on a Windows 7 host).

Within the virtual machine (VM) environment, 4-core SMP was only about 30% faster than single thread rendering. After installing Ubuntu directly onto the computer, rpiece was fast, and the file busy problems disappeared.

Thanks to those who gave advice...I hope this message helps future renderers.

-Chris

···

On 2/1/14 12:03 PM, Chris Kallie wrote:

Thanks Greg and Iebele,

Seems like there are two possible solutions: 1) per Greg's suggestion, monitor and wait for the files to unlock (or kill them); or, 2) create octrees at each iteration (if I'm understanding Iebele's suggestion correctly). My guess is that creating the octrees at the beginning of each iteration gives the OS plenty of time for the unlock to cascade. Is that a reasonable possibility/assumption, or did I misunderstand the second solution?

I will try these (and any other suggestions), and report if successful...

Thanks again!
Chris

On 1/31/14 3:43 PM, Gregory J. Ward wrote:

Let's keep this on radiance-general.

Thanks.
-Greg

Begin forwarded message:

*From: *iebele abel <[email protected] <mailto:[email protected]>>
*Date: *January 31, 2014 12:03:14 PM PST
*To: *High Dynamic Range Imaging <[email protected] <mailto:[email protected]>>
*Subject: **RE: [HDRI] SMP file busy problem on iterative rendering in Linux*

Dear Chris,

I had something similar, using a similar script. That situation - a lock on the pp file - was solved after all octrees were compiled again, although rvu and rpict did not complain. Somehow rpiece locked the pp file, however. Strange... Good luck,

iebele

---- Chris Kallie schreef ----

Dear Radiance Users,

I've assembled a simple script that iterates through a folder containing
octrees, and renders using SMP. On Mac, everything works as expected.
However, on Linux, I get errors suggesting both the syncfile and ppfile
are still locked by the OS. I've attempted two solutions, including
adding 'sleep 5' after pfilt (first attempt, shown below), and then by
externalizing the while loop, with an ending 'exit 0' (second attempt,
not shown here), but neither approach fixed the problem.

Here is one version of code which works on Mac but not in Linux:

for file in obj/*.oct
do
   filename="${file%.*}"
   rm -f syncfile argsfile ppfile
   echo 1 64 > syncfile
   cat syncfile
   echo -F syncfile @arguments -vf room.vf -o $filename.unf
$filename.oct > argsfile
   cat argsfile
   NCPU=2
   while [ "$NCPU" -gt 0 ]
   do
       echo "running proc $NCPU"
       rpiece -v -PP ppfile @argsfile &
       sleep 5
       NCPU=`expr $NCPU - 1`
   done
   wait
   echo "Filtering File - $filename.unf"
   pfilt -1 -e -2 -m .25 -r .6 -x /3 -y /3 $filename.unf > $filename.hdr
   sleep 5
   rm -f syncfile argsfile ppfile
done

Interestingly, I do not get an error saying "cannot remove 'argsfile' ".
After the first image is successfully rendered, I get the following
errors at each successive iteration, indicating the rm only works once
(line 4) in the first iteration:

rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy
rm: cannot remove 'syncfile': Text file busy
rm: cannot remove 'ppfile': Text file busy

Since the error isn't fatal, the machine keeps rendering, and ends up
rendering the same (first) image over and over. Has anyone run into this
problem and found a solution? Any advice would be greatly appreciated.

Thanks,
Chris

_______________________________________________
HDRI mailing list
[email protected] <mailto:[email protected]>
http://www.radiance-online.org/mailman/listinfo/hdri

*From: *[email protected] <mailto:[email protected]>
*Date: *January 31, 2014 12:34:20 PM PST
*Subject: **confirm 20969091e7e2e4da0d2a03094f2222341b41133d*

If you reply to this message, keeping the Subject: header intact,
Mailman will discard the held message. Do this if the message is
spam. If you reply to this message and include an Approved: header
with the list password in it, the message will be approved for posting
to the list. The Approved: header can also appear in the first line
of the body of the reply.

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general