Deterministic calculations with Radiance

Christopher_Kings-Ly · February 2, 2005, 1:13pm

Hi,

I'm thinking about setting up automated regression testing for my Radiance application.

Now, the problem is for these I wish to make Radiance run deterministically, so that I can run diffs of output against known good output, etc.

Is there a way of doing this? Do I need to set a bunch of jitter arguments to 0?

Cheers,

Chris

Compagnon_Raphael · February 2, 2005, 2:05pm

I think these two rpict parameters have to be specified for this task:

-ps 0 -pj 0

···

----- Original Message -----
From: Christopher Kings-Lynne <[email protected]>
Date: Wednesday, February 2, 2005 2:13 pm
Subject: [Radiance-general] Deterministic calculations with Radiance

Hi,

I'm thinking about setting up automated regression testing for my
Radiance application.

Now, the problem is for these I wish to make Radiance run
deterministically, so that I can run diffs of output against known
good
output, etc.

Is there a way of doing this? Do I need to set a bunch of jitter
arguments to 0?

Cheers,

Chris

Greg_Ward · February 2, 2005, 2:21pm

Hi Chris,

If you turn off all stochastic processes in Radiance, you get the worst imaginable results. I don't think there's any way to get repeatable output from Radiance that isn't terrible. To see what I'm talking about, try compiling with the -DNORANDOM flag. Even with this flag set, the results will be hypersensitive to small changes in illumination values, etc., as the decision on whether to trace particular rays may be go above or below the threshold. The only way to avoid this is to set -lw 0 and rely on the -lr parameter to end recursion.

The ideal approach would be to use a pseudo-random number generator that is repeatable. In principle, Radiance does this, but due to differences between machines and implementations, in practice it does not work out that way.

-Greg

···

From: Christopher Kings-Lynne <[email protected]>
Date: February 2, 2005 5:13:27 AM PST

Hi,

I'm thinking about setting up automated regression testing for my Radiance application.

Now, the problem is for these I wish to make Radiance run deterministically, so that I can run diffs of output against known good output, etc.

Is there a way of doing this? Do I need to set a bunch of jitter arguments to 0?

Cheers,

Chris

Randolph_Fritz · February 2, 2005, 5:16pm

Why not incorporate some random number generators with Radiance,
instead of relying on whatever the system happens to provide?

Randolph

···

On Wed, Feb 02, 2005 at 06:21:18AM -0800, Gregory J. Ward wrote:

The ideal approach would be to use a pseudo-random number generator
that is repeatable. In principle, Radiance does this, but due to
differences between machines and implementations, in practice it does
not work out that way.

Greg_Ward · February 3, 2005, 5:40am

I didn't really explain this right. The problem is not just different implementations -- but what I said earlier about thresholded values causing the random number sequence to get off track. There's no easy fix for this, and trying to reproduce exact pixel values will always be a problem with any Monte Carlo renderer. If you don't believe me, try it yourself.

-G

···

From: Randolph Fritz <[email protected]>
Date: February 2, 2005 9:16:58 AM PST

On Wed, Feb 02, 2005 at 06:21:18AM -0800, Gregory J. Ward wrote:

The ideal approach would be to use a pseudo-random number generator
that is repeatable. In principle, Radiance does this, but due to
differences between machines and implementations, in practice it does
not work out that way.

Why not incorporate some random number generators with Radiance,
instead of relying on whatever the system happens to provide?

Randolph

Christopher_Kings-Ly · February 3, 2005, 10:20am

Hmmm, I guess I still don't quite get it...

You are using many calls to random() or something. If you srandom a certain seed, why is it not possible to get the same output every time?

Sorry...

Chris

Gregory J. Ward wrote:

···

I didn't really explain this right. The problem is not just different implementations -- but what I said earlier about thresholded values causing the random number sequence to get off track. There's no easy fix for this, and trying to reproduce exact pixel values will always be a problem with any Monte Carlo renderer. If you don't believe me, try it yourself.

-G

From: Randolph Fritz <[email protected]>
Date: February 2, 2005 9:16:58 AM PST

On Wed, Feb 02, 2005 at 06:21:18AM -0800, Gregory J. Ward wrote:

The ideal approach would be to use a pseudo-random number generator
that is repeatable. In principle, Radiance does this, but due to
differences between machines and implementations, in practice it does
not work out that way.

Why not incorporate some random number generators with Radiance,
instead of relying on whatever the system happens to provide?

Randolph

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general

Greg_Ward · February 3, 2005, 2:53pm

Say you are tracing your 10,000,000 rays to generate a test image that you hope to compare to the official result for Radiance 3.6.1. Somewhere in the middle of your image (more likely the beginning), you hit some surface deep in the ray tree after 4 or 5 reflections, and Radiance makes the following test:

if (r->rweight < minweight)
return(0); /* don't trace this ray */

where minweight is set by the -lw option to rpict. Well, it just so happens that r->rweight is almost exactly equal to minweight at this point in the calculation. On one machine, the statement will evaluate to true, and on another machine architecture, it evaluates to false. Big deal. You've just altered the result by maybe 1e-3, which is within the tolerance of your image comparison metric.

However, since you've decided not to trace that ray and all the rays that spawn from it, let's say that's 12 calls to random() that you won't be making during this rendering. (It doesn't matter how many calls it is you miss -- one is enough.) All your subsequent calls to random() will therefore be completely different from what they were on that other machine, even using the exact same implementation of random().

How likely is the above scenario? Nearly 100%, because in 10,000,000 rays you are bound to find some threshold test somewhere in the code that can go one way or another. Setting -lw 0 is only a partial solution, because there are other floating point tests and thresholds that can't be altered or avoided within the code. Welcome to the vaguaries of floating-point calculations.

-Greg

···

From: Christopher Kings-Lynne <[email protected]>
Date: February 3, 2005 2:20:25 AM PST

Hmmm, I guess I still don't quite get it...

You are using many calls to random() or something. If you srandom a certain seed, why is it not possible to get the same output every time?

Sorry...

Chris

Christopher_Kings-Ly · February 3, 2005, 3:51pm

Say you are tracing your 10,000,000 rays to generate a test image that you hope to compare to the official result for Radiance 3.6.1.

Ah, what I meant is comparing ourselves to ourselves, on the same PC...

Would that be doable?

Chris

Tarik_Rahman · February 3, 2005, 4:13pm

Hi there
I'm trying this line from a previous thread for info about rays in an image
ximage someRendering.pic rtrace -ab 5 -oodLpnsm -h -x 1 -i someOct.oct

I get the image displayed but these lines appear
./ximage: rtrace : bad picture format
./ximage: -ab : cannot open picture file
./ximage: 5 : cannot open picture file
./ximage: -oodLpnsm : cannot open picture file
./ximage: -h : cannot open picture file
./ximage: -x : cannot open picture file
./ximage: 1 : cannot open picture file
./ximage: -i : cannot open picture file
./ximage: testroom.oct : bad picture format

left clicking and dragging on the image makes a rectangle selection but nothing
happens
right clicking I get just the origin and direction outputted

I know I must be doing something simple, wrong

Thanks for any help

Tarik

···

--
Tarik Rahman
PhD student, Institute of Perception, Action and Behaviour
School of Informatics
University of Edinburgh

Andrew_McNeil1 · February 3, 2005, 4:23pm

Looks like you're missing the pipe symbol.

Try:
ximage someRendering.pic | rtrace -ab 5 -oodLpnsm -h -x 1 -i someOct.oct

The "|" is typed by pressing shift and backshash on a US keyboard. If you
are unfamiliar with pipes check out these links for an explanation:
http://www.december.com/unix/tutor/pipesfilters.html
http://unix.t-a-y-l-o-r.com/USpipes.html

···

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Tarik
Rahman
Sent: Thursday, February 03, 2005 11:13 AM
To: Radiance general discussion
Subject: Re: [Radiance-general] Deterministic calculations with Radiance

Hi there
I'm trying this line from a previous thread for info about rays in an image
ximage someRendering.pic rtrace -ab 5 -oodLpnsm -h -x 1 -i someOct.oct

I get the image displayed but these lines appear
./ximage: rtrace : bad picture format
./ximage: -ab : cannot open picture file
./ximage: 5 : cannot open picture file
./ximage: -oodLpnsm : cannot open picture file
./ximage: -h : cannot open picture file
./ximage: -x : cannot open picture file
./ximage: 1 : cannot open picture file
./ximage: -i : cannot open picture file
./ximage: testroom.oct : bad picture format

left clicking and dragging on the image makes a rectangle selection but
nothing happens right clicking I get just the origin and direction outputted

I know I must be doing something simple, wrong

Thanks for any help

Tarik

--
Tarik Rahman
PhD student, Institute of Perception, Action and Behaviour School of
Informatics University of Edinburgh

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general

___________________________________________________________________
Electronic mail messages entering and leaving Arup business
systems are scanned for acceptability of content and viruses.

Greg_Ward · February 4, 2005, 7:09am

Yes, but what's the point? A change in the compiler or a minor alteration of the code that makes no difference to the results but affects the order in which rays are traced will completely defeat such a test. How can we do regression testing with something that unstable?

-Greg

···

From: Christopher Kings-Lynne <[email protected]>
Date: February 3, 2005 7:51:36 AM PST

Say you are tracing your 10,000,000 rays to generate a test image that you hope to compare to the official result for Radiance 3.6.1.

Ah, what I meant is comparing ourselves to ourselves, on the same PC...

Would that be doable?

Chris

Randolph_Fritz · February 4, 2005, 5:48pm

Could we perhaps "blur" the two light maps before comparing them?

Randolph

···

On Thu, Feb 03, 2005 at 11:09:33PM -0800, Greg Ward wrote:

Yes, but what's the point? A change in the compiler or a minor
alteration of the code that makes no difference to the results but
affects the order in which rays are traced will completely defeat such
a test. How can we do regression testing with something that unstable?

Greg_Ward · February 4, 2005, 6:04pm

Perhaps. If the pixel sampling rate were high enough (e.g. pfilt -x /10 -y /10), you might get some stability. At that point, you shouldn't have to worry about the random number generator. However, the MC ray tracing literature is full of examples where even 200+ samples/pixel is not enough to avoid large variance in some regions, and the same can be true for Radiance, especially with specular samples.

-G

···

From: Randolph Fritz <[email protected]>
Date: February 4, 2005 9:48:23 AM PST

On Thu, Feb 03, 2005 at 11:09:33PM -0800, Greg Ward wrote:

Yes, but what's the point? A change in the compiler or a minor
alteration of the code that makes no difference to the results but
affects the order in which rays are traced will completely defeat such
a test. How can we do regression testing with something that unstable?

Could we perhaps "blur" the two light maps before comparing them?

Randolph

Randolph_Fritz · February 10, 2005, 12:32am

Perhaps with carefully chosen models and parameters? I would expect
it to be possible to compare well-chosen test scenes with appropriate
parameters between versions, as well as with actual physical models,
so long as a random number generator of reasonable quality is used.
Seems to me such combinations of scenes and parameters would be
excellent testing and debugging tools. (Having the physical models
for comparison would be *most* excellent, but let us not assume there
is a budget for them!)

Randolph, still drowning here in Portland, Oregon.

···

On Fri, Feb 04, 2005 at 10:04:19AM -0800, Greg Ward wrote:

Perhaps. If the pixel sampling rate were high enough (e.g. pfilt -x
/10 -y /10), you might get some stability. At that point, you
shouldn't have to worry about the random number generator. However,
the MC ray tracing literature is full of examples where even 200+
samples/pixel is not enough to avoid large variance in some regions,
and the same can be true for Radiance, especially with specular
samples.

-G

>From: Randolph Fritz <[email protected]>
>Date: February 4, 2005 9:48:23 AM PST
>
>On Thu, Feb 03, 2005 at 11:09:33PM -0800, Greg Ward wrote:
>>Yes, but what's the point? A change in the compiler or a minor
>>alteration of the code that makes no difference to the results but
>>affects the order in which rays are traced will completely defeat such
>>a test. How can we do regression testing with something that
>>unstable?
>
>Could we perhaps "blur" the two light maps before comparing them?
>
>Randolph

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general

Greg_Ward · February 10, 2005, 7:13am

Yes, I could see doing this. We would of course be accepting differences between versions and machines to some tolerance, but it should be sufficient for spotting any major regressions in the code or between releases. The trick then is to have enough well-designed test scenes to cover all of the basic and some of the more esoteric features of Radiance. (The "prism" and "BRTDfunc" primitives spring to mind.)

-Greg

···

From: Randolph Fritz <[email protected]>
Date: February 9, 2005 4:32:13 PM PST

Perhaps with carefully chosen models and parameters? I would expect
it to be possible to compare well-chosen test scenes with appropriate
parameters between versions, as well as with actual physical models,
so long as a random number generator of reasonable quality is used.
Seems to me such combinations of scenes and parameters would be
excellent testing and debugging tools. (Having the physical models
for comparison would be *most* excellent, but let us not assume there
is a budget for them!)

Randolph, still drowning here in Portland, Oregon.

Christopher_Kings-Ly · February 10, 2005, 9:38am

Yes, I could see doing this. We would of course be accepting differences between versions and machines to some tolerance, but it should be sufficient for spotting any major regressions in the code or between releases. The trick then is to have enough well-designed test scenes to cover all of the basic and some of the more esoteric features of Radiance. (The "prism" and "BRTDfunc" primitives spring to mind.)

I am a developer on the PostgreSQL database, and we have a full suite of regression tests. We use diff to compare actual vs. expected results. Perhaps we could borrow that code.

We have per-platform floating point files that we need to use to allow for differences in IEEE implementation between some platforms. ie. -0, etc.

I was personally thinking for my own project checking that as I make changes to this big script I have, I don't affect other stuff. However, having a regression suite for Radiance seems like something that's way overdue for a piece of software that needs to be correct for legal purposes

Chris

Greg_Ward · February 10, 2005, 4:15pm

I don't think repurposing PostgreSQL code is going to be useful in this case. We'll be comparing floating point values within some delta, probably over large images, and the existing tools should be sufficient for that (pcomb, etc.) We just need to create a few dozen test scenes, a Makefile for generating results, and a generic script for comparing them against the official results for each regression.

It's probably a 2-4 weeks effort to do it right. It's always a question of who has the time, I suppose. We could make assignments, each volunteer testing out their favorite aspect of Radiance. Anyone want to step forward?

-Greg

···

From: Christopher Kings-Lynne <[email protected]>
Date: February 10, 2005 1:38:48 AM PST

Yes, I could see doing this. We would of course be accepting differences between versions and machines to some tolerance, but it should be sufficient for spotting any major regressions in the code or between releases. The trick then is to have enough well-designed test scenes to cover all of the basic and some of the more esoteric features of Radiance. (The "prism" and "BRTDfunc" primitives spring to mind.)

I am a developer on the PostgreSQL database, and we have a full suite of regression tests. We use diff to compare actual vs. expected results. Perhaps we could borrow that code.

We have per-platform floating point files that we need to use to allow for differences in IEEE implementation between some platforms. ie. -0, etc.

I was personally thinking for my own project checking that as I make changes to this big script I have, I don't affect other stuff. However, having a regression suite for Radiance seems like something that's way overdue for a piece of software that needs to be correct for legal purposes

Chris