Fun with multiple cores and rtrace

I finally managed to get Linux going on my modelling system, an eight-core
ThinkStation. It was running Radiance very fast. Then I started seeing ps
output like this:

6589 ? S 0:00 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6592 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6593 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6594 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6595 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6596 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6597 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6598 ? S 41:03 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6599 ? S 49:48 rtrace -n 8 -h -I @sensparm.opt ot-3.oct

And it slowed down by a factor of three. My impression is that there has been an
OS glitch and it is in fact only using one core. Anyone else seen this?

Head Debian build of Radiance, Ubuntu 10.10, most current version.

Randolph

Hi Randolph,

What is unexpected about this process status? You asked for -n 8, so you get 8 active processes and one that collects the results. They are all duplicates of each other. What are the exact circumstances of the slowdown? What options are hidden in your sensparm.opt file?

-Greg

···

From: "Randolph M. Fritz" <[email protected]>
Date: February 2, 2011 9:29:55 PM PST

I finally managed to get Linux going on my modelling system, an eight-core
ThinkStation. It was running Radiance very fast. Then I started seeing ps
output like this:

6589 ? S 0:00 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6592 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6593 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6594 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6595 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6596 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6597 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6598 ? S 41:03 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6599 ? S 49:48 rtrace -n 8 -h -I @sensparm.opt ot-3.oct

And it slowed down by a factor of three. My impression is that there has been an
OS glitch and it is in fact only using one core. Anyone else seen this?

Head Debian build of Radiance, Ubuntu 10.10, most current version.

Randolph

Sensparm.opt attached. What's striking is the way some of the rtrace processes seem to "get ahead" of the others. It was actually worse before I took that ps snapshot, with the last two processes having accumulated run times of 30 minutes or so, while the first six had only run about 10 minutes. The same sensparm.opt file (modulo a change of a -sj) continues to work as expected on dino. The main things I changed seems to have been installing os updates and bringing Radiance to its head version, but I am actually suspecting an os glitch.

Randolph

sensparm.opt:
-dp 2048
-ar 49
-ms 1.9
-ds .2
-dt .05
-dc .75
-dr 3
-sj 1
-st .01
-ab 5
-aa .075
-ad 4096
-as 2048
-av 0.01 0.01 0.01
-lr 12
-lw .0005

···

On 2011-02-02 22:14:52 -0800, Greg Ward said:

Hi Randolph,

What is unexpected about this process status? You asked for -n 8, so you get 8 active processes and one that collects the results. They are all duplicates of each other. What are the exact circumstances of the slowdown? What options are hidden in your sensparm.opt file?

-Greg

From: "Randolph M. Fritz" <[email protected]>
Date: February 2, 2011 9:29:55 PM PST

I finally managed to get Linux going on my modelling system, an eight-core
ThinkStation. It was running Radiance very fast. Then I started seeing ps
output like this:

6589 ? S 0:00 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6592 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6593 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6594 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6595 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6596 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6597 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6598 ? S 41:03 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6599 ? S 49:48 rtrace -n 8 -h -I @sensparm.opt ot-3.oct

And it slowed down by a factor of three. My impression is that there has been an
OS glitch and it is in fact only using one core. Anyone else seen this?

Head Debian build of Radiance, Ubuntu 10.10, most current version.

Randolph

--
Randolph M. Fritz • [email protected]
Environmental Energy Technologies Division • Lawrence Berkeley Labs

Hi Randolph,

Not sure if this is related, but I recall that Rob Guglielmetti looked into rtrace performance in a parallel setting:

http://www.rumblestrip.org/interests/light/rtrace-multiprocessing-option-initial-test-results/

-Jack

···

--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction

On 2/3/2011 11:49 AM, Randolph M. Fritz wrote:

Sensparm.opt attached. What's striking is the way some of the rtrace processes seem to "get ahead" of the others. It was actually worse before I took that ps snapshot, with the last two processes having accumulated run times of 30 minutes or so, while the first six had only run about 10 minutes. The same sensparm.opt file (modulo a change of a -sj) continues to work as expected on dino. The main things I changed seems to have been installing os updates and bringing Radiance to its head version, but I am actually suspecting an os glitch.

Randolph

sensparm.opt:
-dp 2048
-ar 49
-ms 1.9
-ds .2
-dt .05
-dc .75
-dr 3
-sj 1
-st .01
-ab 5
-aa .075
-ad 4096
-as 2048
-av 0.01 0.01 0.01
-lr 12
-lw .0005

On 2011-02-02 22:14:52 -0800, Greg Ward said:

Hi Randolph,

What is unexpected about this process status? You asked for -n 8, so you get 8 active processes and one that collects the results. They are all duplicates of each other. What are the exact circumstances of the slowdown? What options are hidden in your sensparm.opt file?

-Greg

From: "Randolph M. Fritz" <[email protected]>
Date: February 2, 2011 9:29:55 PM PST

I finally managed to get Linux going on my modelling system, an eight-core
ThinkStation. It was running Radiance very fast. Then I started seeing ps
output like this:

6589 ? S 0:00 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6592 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6593 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6594 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6595 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6596 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6597 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6598 ? S 41:03 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6599 ? S 49:48 rtrace -n 8 -h -I @sensparm.opt ot-3.oct

And it slowed down by a factor of three. My impression is that there has been an
OS glitch and it is in fact only using one core. Anyone else seen this?

Head Debian build of Radiance, Ubuntu 10.10, most current version.

Randolph

Hi Randolph,

We probably shouldn't jump to conclusions about CPU utilization based on how much time is attributed to each rtrace process. Due to the way rays get assigned to processes as they finish what they've got, one rtrace process might end up using more time than another that had to wait a longer for its assignment.

One thing is certain: you should be sharing an ambient file via the -af option if -aa is non-zero and -ab is positive. Otherwise, your process are mostly wasting time computing values that other processes have already computed. Rob found the same thing in the stats Jack pointed out.

Cheers,
-Greg

···

From: "Randolph M. Fritz" <[email protected]>
Date: February 3, 2011 8:49:30 AM PST

Sensparm.opt attached. What's striking is the way some of the rtrace processes seem to "get ahead" of the others. It was actually worse before I took that ps snapshot, with the last two processes having accumulated run times of 30 minutes or so, while the first six had only run about 10 minutes. The same sensparm.opt file (modulo a change of a -sj) continues to work as expected on dino. The main things I changed seems to have been installing os updates and bringing Radiance to its head version, but I am actually suspecting an os glitch.

Randolph

sensparm.opt:
-dp 2048
-ar 49
-ms 1.9
-ds .2
-dt .05
-dc .75
-dr 3
-sj 1
-st .01
-ab 5
-aa .075
-ad 4096
-as 2048
-av 0.01 0.01 0.01
-lr 12
-lw .0005

On 2011-02-02 22:14:52 -0800, Greg Ward said:

Hi Randolph,
What is unexpected about this process status? You asked for -n 8, so you get 8 active processes and one that collects the results. They are all duplicates of each other. What are the exact circumstances of the slowdown? What options are hidden in your sensparm.opt file?
-Greg

From: "Randolph M. Fritz" <[email protected]>
Date: February 2, 2011 9:29:55 PM PST
I finally managed to get Linux going on my modelling system, an eight-core
ThinkStation. It was running Radiance very fast. Then I started seeing ps
output like this:
6589 ? S 0:00 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6592 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6593 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6594 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6595 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6596 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6597 ? R 29:32 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6598 ? S 41:03 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
6599 ? S 49:48 rtrace -n 8 -h -I @sensparm.opt ot-3.oct
And it slowed down by a factor of three. My impression is that there has been an
OS glitch and it is in fact only using one core. Anyone else seen this?
Head Debian build of Radiance, Ubuntu 10.10, most current version.
Randolph

you should be sharing an ambient file via the -af option if -aa is non-zero and -ab is positive.

D'oh! I'll fix that.

Jack de Valpine:

http://www.rumblestrip.org/interests/light/rtrace-multiprocessing-option-initial-test-results/

Yes,

I think that will be very useful.

Thank you all.

···

On 2011-02-03 09:18:30 -0800, Greg Ward said:
--
Randolph M. Fritz • [email protected]
Environmental Energy Technologies Division • Lawrence Berkeley Labs