A first look at scons

radiance-dev
1

I decided to put my money where my mouth was, and try to make scons build the Radiance libraries in "src/common".

In general, "scons" works fairly well. Starting from a general knowlege of Python, it took me perhaps eight hours of work to learn scons well enough to assemble an overall configuration script, and one which builds the common libraries.

Scons uses a directory hierarchy model. At the top of the hierarchy, a file named, by default, "SConstruct" is used, which sets up the build, determining system configuration, and including the various build files (by convention named "SConscript") from subdirectories. To build the whole thing, one then uses the command "scons" in the top-level directory. To build parts, one uses "scons <subdirectory>" to build a subdirectory's files from the top or "scons -u" from within a subdirectory.

Scons uses a two-phase build model, where a description of the build is constructed first, and then targets are built. The build description is constructed by the Python script, SConstruct, which includes the subsidiary SConscript scripts. The scripts are themselves run in a special environment provided by scons which includes three special classes: one of describing build environments, one for describing command-line options, and one for testing the build system configuration. The build portion is impressively well-worked out, and automatically includes dependencies for many languages--there is never a need to specify dependencies on .h files, as far as I can tell.

What I have is a global configuration script, and a script that describes the build of the common library. They are attached, and you can read them and try them. I think they're pretty straightforward.

So far, I think this is a winner. There are some rough spots--forgetting to use Python lists in certain places can produce amazingly Python-esque messages, but, those being identified, it works fairly well. Keep in mind I've yet to build and install any actual executables--I may change my mind once I have.

Files are attached--unless someone screams "NO! NO!" my next step will be to write a build and install script for the programs in src/meta.

Randolph

SConstruct (2.42 KB)

SConscript (1.23 KB)

2

Randolph Fritz wrote:

I decided to put my money where my mouth was, and try to make scons
build the Radiance libraries in "src/common".

[...]

What I have is a global configuration script, and a script that
describes the build of the common library. They are attached, and you
can read them and try them. I think they're pretty straightforward.

Nice start. I played around with this a bit, added a SConscript
file for src/rt, and after some more fiddling, I ended up with
the following:

- If you use triple quotes (''' or """), then you don't need to
  escape line wraps in literal Python strings.

- I changed the imports of the standard modules to just import
  the module. It is usually less confusing to explicitly refer
  eg. to the fully qualified sys.platform in the code, especially
  for such generic names.

- ("this") is just "this" in Python. But I think you already
  figured this out on your own...

- You created one environment named "env". I changed that name to
  "build", so that we can later establish a seperate "install"
  target (those names are used when you do eg. an "scons install").
  There should also be a way to have "install" depend on "build".

- Check what the man page says about opts.Update(). I don't think
  it really makes a functional difference, but what they
  recommend is probably better style and may have other
  advantages down the road (not changed yet).

- The compile options flag is named CCFLAGS, not CFLAGS.

- I changed the configure routines to look for the GL library
  like you already did with X11, and not just for the headers.
  Then I moved those routines to a seperate module
  build_utils/find_libs.py. Maybe the option related routines can
  go into build_utils as well.

- When checking for X11, I don't include the resulting paths in
  the global CPPPATH and LIBPATH. It seems better to store them
  in seperate variables, and add them as overrides to only those
  files that actually need them. You can then also skip building
  eg. rview if the respective variables aren't set.

- Similarly, I pass [NO]STEREO and SPEED as seperate variables,
  which may be used to override the defaults further down.

- I have named all Radiance specific variables "RAD_XXX", so that
  there will never be any confusion.

- In src/common/SConscript, you can use "#src/lib/rt" as a target
  for the library, so it will be built in the correct place.

- Adding a library to env['LIBS'] means that every single
  executable will be linked with it after that. I'm sure that a
  majority currently does use librt.a, but maybe we can be more
  specific here to improve the performance of the linking.
  In src/rt, I'm making a copy of the "build" environment with
  the changes required for all the targets there.

- The env['PLATFORM'] value is tuned for internal use by SCons.
  I have decided to use the more standard sys.platform (and
  os.name, where that isn't enough) for our own purposes.
  SConstruct now supposts (almost) the full range of settings
  from makeall.

  If that still isn't enough, we also have os.uname(), at least
  when os.name == 'posix'. This may be necessary eg. in case we
  need to distinguish between Solaris or Linux on different
  hardware.

- I moved unix_process.c resp. win_process.c to a variable named
  "RAD_PROCESS", which then gets added to the right file list in
  src/common. Other files like "win_popen.c" may follow.

- I'll have to experiment a bit more with automatically
  generating C files from a Python function, so that we can
  replace the current "ed" script in a platform independent way.

- Interestingly, SCons uses c++ to link the executables on my
  system. I think they do so to improve flexibility, in case
  there might be any C++ object files involved. At first, I was a
  bit confused by that, but I couldn't find any disadvantages of
  this approach.

The results of my fiddling are attached as a tgz, which expands
directly into the build tree from ray/.

Some of my experiments uncovered general build issues in Radiance
(eg. duplicate function names), which I'll address seperately.
I did build the binaries in src/rt (and everything they depend
on), and it worked very nicely.

I'm inclined to check this into CVS, so that we can use the
revision management there. For the moment, I don't think it will
interfere in any way with the old build system, so that a
coexistence of the two shouldn't be a problem.

-schorsch

raysc02.tgz (3.6 KB)

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/

3

Hi Schorsch,

This looks pretty cool. I have the following comments:

1) The overall SConstruct file doesn't look that much smaller than makeall, though it is better organized. It's missing some things, like the license agreement, which must be added, and the ability to edit the build options. Once you add these, assuming you can, it will probably be about the same size as makeall, which is OK.

2) However, I'm wondering if it's even possible with this system to allow the people doing the compiles to mess with the options without editing the SConstruct script, and therefore understanding it. I wouldn't feel comfortable editing this myself, as Python is a whole new, foreign environment to me.

3) If you added a way to change the build options, would SCons then rebuild all the *.o files automatically? What's the "clean" process? Makeall has a little test to see if the rmake script has been updated and does a clean if it has. Can we include such functionality with SCons?

4) Since SCons is figuring out all the dependencies for us, which by the way is unnecessary for end users, who build the whole system from scratch and that's about all they do, I'm wondering how these dependencies are worked out? Does it look in all the source files every time to find the headers? Seems a bit time consuming if all you're doing is rebuilding after a single change, but I haven't seen it work so I don't actually know if it slows down the process or not.

5) Personally, I like the process we currently have of building the rmake script, which is just a call to make with a bunch of options, and building separately. Does combining the two with a whole environment of passed variables means you can only recompile at the top level, or is there some way to build in a subdirectory only? If it was wasteful to refigure the dependencies in just one directory, refiguring them over the entire system each time you compile a small change has GOT to be a factor, doesn't it?

Well, those are my concerns for now. I really appreciate you and Fritz looking into this.
-Greg

4

Greg Ward wrote:

1) The overall SConstruct file doesn't look that much smaller than
makeall, though it is better organized. It's missing some things, like
the license agreement, which must be added, and the ability to edit the
build options. Once you add these, assuming you can, it will probably
be about the same size as makeall, which is OK.

The platform configs and the license statement are the biggest
chunks in there, so that's no surprise...

2) However, I'm wondering if it's even possible with this system to
allow the people doing the compiles to mess with the options without
editing the SConstruct script, and therefore understanding it.

It's fairly simple as it is now, but we can make it as nice as we
want. Remeber, we now have a "real" programming language at our
disposal at that point. For example, Python brings a module for
parsing the traditional configuration file format:

  [build]
  CC = cc
  CPPFLAGS = -DBSD -Dfreebsd
  CCFLAGS = -O2
  [install]
  bindir = /opt/radiance3.6a/bin
  sharedir = /opt/radiance3.6a/share
  mandir = /opt/radiance3.6a/man

This could be used for both the shipped platform specific
configuration files, as well as for storing whatever the user has
chosen to modify. There may be other ways to shortcut a few tests
on a box where we have run before.

3) If you added a way to change the build options, would SCons then
rebuild all the *.o files automatically?

Scons will automatically rebuild everything where either the
build options or the dependencies have changed.

What's the "clean" process?

scons -c

Makeall has a little test to see if the rmake script has been updated
and does a clean if it has. Can we include such functionality with
SCons?

Not needed here.

4) Since SCons is figuring out all the dependencies for us, which by
the way is unnecessary for end users, who build the whole system from
scratch and that's about all they do, I'm wondering how these
dependencies are worked out? Does it look in all the source files
every time to find the headers? Seems a bit time consuming if all
you're doing is rebuilding after a single change, but I haven't seen it
work so I don't actually know if it slows down the process or not.

Good questions... I find it surprisingly fast considering all the
information it has to look at. I think there's a fairly smart
combination of timestamps, cryptographic hashes, and dependency
caching at work. All the bells and whistles may not matter for
someone who just compiles everything once, but the automatic
dependencies can save a lot of time for developers making small
changes here and there. Those benefits will probably increase
when we split librt.a into several smaller units, because changes
under common will have less widespread consequences then.

5) Personally, I like the process we currently have of building the
rmake script, which is just a call to make with a bunch of options, and
building separately. Does combining the two with a whole environment
of passed variables means you can only recompile at the top level, or
is there some way to build in a subdirectory only?

To compile just the stuff in one directory:
in ray: "sconf src/common"
in src/common: "sconf -u"

To compile just one file:
in ray: "sconf src/common/cone.o"
in src/common: "sconf -u cone.o"

Other than make, it will also update any dependencies outside of
the current directory when necessary.

If it was wasteful
to refigure the dependencies in just one directory, refiguring them
over the entire system each time you compile a small change has GOT to
be a factor, doesn't it?

Obviously, all that power requires some overhead. Reading all the
files and analyzing the dependencies takes a few seconds on my
old 300 Mhz box. But since modern hardware is almost ten times as
fast, I don't expect that to be a problem. I'm also not sure if we
already have an optimal setup. We'll only find out by trying...

-schorsch

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/

5

1) The overall SConstruct file doesn't look that much smaller than
makeall, though it is better organized. It's missing some things, like
the license agreement, which must be added, and the ability to edit the
build options. Once you add these, assuming you can, it will probably
be about the same size as makeall, which is OK.

Good. I'm working on some of the build option stuff.

2) However, I'm wondering if it's even possible with this system to
allow the people doing the compiles to mess with the options without
editing the SConstruct script, and therefore understanding it. I
wouldn't feel comfortable editing this myself, as Python is a whole
new, foreign environment to me.

Options can be given on the command line; I think the first version I
sent out even had some help in it; try "scons -h". All the options
aren't in yet, however. On the other hand, it searches out the X and
OpenGL libraries, which simplifies the script. I think it's possible
to check for a native libtiff as well.

3) If you added a way to change the build options, would SCons then
rebuild all the *.o files automatically? What's the "clean" process?
Makeall has a little test to see if the rmake script has been updated
and does a clean if it has. Can we include such functionality with
SCons?

"scons -c" is the clean command. "Rebuild all" on changed options is
possible; I didn't do it in the first version.

4) Since SCons is figuring out all the dependencies for us, which by
the way is unnecessary for end users, who build the whole system from
scratch and that's about all they do, I'm wondering how these
dependencies are worked out? Does it look in all the source files
every time to find the headers? Seems a bit time consuming if all
you're doing is rebuilding after a single change, but I haven't seen it
work so I don't actually know if it slows down the process or not.

A bit of both; it caches (by default) MD5 checksums to check if files
are changed, I think it only scans files that have been changed for
headers.

5) Personally, I like the process we currently have of building the
rmake script, which is just a call to make with a bunch of options, and
building separately. Does combining the two with a whole environment
of passed variables means you can only recompile at the top level, or
is there some way to build in a subdirectory only? If it was wasteful
to refigure the dependencies in just one directory, refiguring them
over the entire system each time you compile a small change has GOT to
be a factor, doesn't it?

"scons src/common" or (cd src/common; scons -u). I don't know what
rules it uses for rescanning in that case, but it doesn't do that much
rescanning most of the time.

I've got some more stuff in, unfortunately I went in a somewhat
different direction than Schorsch. I'll send the new versions out &
people can have a look.

Randolph

···

On Sat, Aug 09, 2003 at 09:02:58AM -0700, Greg Ward wrote: