SCons/Windows status

Hi all,

I think I figured out why xform (and others) had troubles with
multiline !commands on Windows in my earlier attempts. The thing
is, that cmd.com doesn't handle "\\\n" escapes. I now redirected
all calls to popen() to my own win_peopen(). This one correctly
emulates the "|" handling of unix shells, and removes any
disturbing character sequences before invoking the external
programs through Windows native tools. At the moment, it only
supports mode "r", though, which means that igraph, plot4,
ra_rgbe, rad, and rlux will fail during runtime, even if they
successfully compile.

I added a gethomedir.c for posix and Windows, which simplifies
the internals of getpath.c.

I added a util/win_netproc.c, which I found in the Radiance 3.2
sources. It actually only defines stubs to most of the required
functions, so that ranimate will compile but may not do anything
useful (haven't tried). All in all we now can compile 111
executables on Windows, including the "special" targets.

I updated the SCons files, so that it is now possible to create a
debug build with the commandline argument DEBUG=1. This will be
remembered, so you need to use DEBUG=0 for your next release
build after that. I'm more and more leaning towards the idea of
building into seperate trees for release/debug builds and
differnt platforms. As long the build paths remain constant, a
switch between release and debug mode will automatically rebuild
all affected dependencies.

As an experiment, I reconfigured SCons to create several smaller
libraries instead of one big librt.a from the common directory.
The currently resulting number of libraries may be excessive, but
I wanted to see how far we can go. It's likely that the final
splitting will look different. Turns out that having two build
systems next to each other really helps for this kind of testing.

As another positive consequence, it seems that the duplicate file
and function names throughout Radiance don't prevent us from
building stuff on Windows anymore. We should probably still
eliminate most of those redundancies, but that's not quite as
urgent anymore as it first looked like.

Here's the current setup, with files and dependencies for
each library:

librterror.a
  eputs.c
  error.c
  quit.c
  wputs.c
  ->

librtmem.a # allocate and free memory
  bmalloc.c
  ealloc.c
  -> error

librtcont.a # containers: generic in-memory storage and retreival
  savestr.c
  savqstr.c
  lookup.c
  -> error

librtmath.a
  fvect.c
  invmat4.c
  linregr.c
  mat4.c
  tcos.c
  urand.c
  urind.c
  zeroes.c
  dircode.c
  erf.c
  clip.c
  multisamp.c
  plocate.c
  -> error

librtfunc.a
  biggerlib.c
  caldefn.c
  calexpr.c
  calfunc.c
  calprnt.c
  chanvalue.c
  -> error
  -> math (error)
  -> mem (error)
  -> cont (error)

librtio.a # reading from and writing to files
  fdate.c
  fgetline.c
  fgetval.c
  fgetword.c
  fputword.c
  loadvars.c
  portio.c
  wordfile.c
  words.c
  header.c
  -> (frexp.c)
  -> mem (error)
  -> error

librtargs.a # process arguments within processes
  badarg.c
  expandarg.c
  fixargv0.c
  -> mem (error)
  -> io (mem error)

librtpic.a # picture file handling
  color.c
  colrops.c
  resolu.c
  image.c
  -> math (error)
  -> mem (error)
  -> io (mem error)
  -> args (io mem error)

librtcolor.a # color math
  spec_rgb.c
  ->

tonemap # picture tone mapping
(this is actually included in librtpic.a at the moment)
  tmapcolrs.c
  tmapluv.c
  tmaptiff.c
  tmap16bit.c
  tonemap.c
  -> color
  -> pic (math mem io error)
  -> tiff
  -> io (mem error)

librtpath.a # manipulating file names and files (but not the contents)
  getpath.c
  gethomedir.c
  paths.c
  getlibpath.c
  -> io (string manipulation)

librtproc.a # processes: call them, feed them, kill them
  process.c
  unix_process.c
  win_process.c
  win_popen.c
  -> paths
  -> io (error)

librtnet.a
  myhostname.c
  ->

librtscene.a # scene and octree files
  cone.c
  face.c
  font.c
  mesh.c
  instance.c
  tmesh.c
  sceneio.c
  readobj.c
  modobject.c
  objset.c
  readfargs.c
  otypes.c
  free_os.c
  addobjnotify.c
  octree.c
  readmesh.c
  readoct.c
  xf.c (args/math.h)

librgl.a # OpenGL
  rgldomat.c
  rglfile.c
  rglinst.c
  rglmat.c
  rglsrc.c
  rglsurf.c
  -> error
  -> mem (error)
  -> io (mem error)
  -> math (error)
  -> sys
  -> scene/octree
  -> xf.c

librtlamps.a
  lamps.c
  rexpr.c
  fropen.c [use getpath.c instead?]
  -> color

stuff that gets used to replace system functions on some
platforms:
  popen.c
  mktemp.c
  strcmp.c
  preadwrite.c
  frexp.c
  fwrite.c
  random.c
  malloc.c -> getpagesize.c
    [getpagesize.c]

unused
  bcopy.c
  peano.c

Comments:

The most obvous candidate for merging are librtcont.a and
librtmem.a. Although there are several programs that only use one
of them, they are related and small enough to be packed together.

I'm not sure how tricky it would be to split up librtio.a
further, to seperate plain string manipulation from those
functions that are actually used to access files. I don't know
yet whether that would make sense either.

Yes, librtcolor.a is very small. It's used seperately of
librtpic.a several times, as a dependency of librtlamps.a.

The tonemapping could justifiably be seperated from librtpic.a,
there are only dependencies in one direction.

Librtargs.a and librtpath.a both provide rather specialized
functionality, which depends on the string functions (and only
those) from librtio.a.

Librtnet.a may eventually grow, when (and if) we add remote
ambient file access in some way or another. It might be worth
considering to merge librtpath.a, librtproc.a, and librtnet.a
into a librtsys.a. Those three libraries all have a lot of OS
dependencies to deal with, and they are likely to be used in
combination.

I didn't find the time and energy yet to analyze all the
dependencies within librtscene.a yet, but it probably depends on
almost anything else.

There are two identical copies of x11findwind.c around. Maybe
we can find a few more files that could be combined into a
librtx11.a?

When did anyone check which of the OS replacement functions are
really necessary anymore? Some of them seem to be written to
replace really old and buggy implementations that may have gone
out of use by now anyway.

Can we remove those files that aren't used at all?

The parts that currently still don't compile on Windows are
everything to do with the tifflib, the holodeck, and the GUI
programs, as well as makedist and ranimove (the latter two
because they use fork()/exec() and friends).

Oh, you're still reading that far down? Congrats! :wink:

-schorsch

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/

Hi Georg,

Thanks for doing all this work on the Windows port -- great progress! I'm a little confused as to the benefit(s) of separating the objects into so many libraries. Is this just to help reduce memory requirements for dynamic linking? On Unix at least, it complicates the builds considerably, as you need to know the correct link order for the libraries or you get undefined symbol errors at load time.

From: Georg Mischler <[email protected]>
Date: Mon Oct 27, 2003 6:43:46 AM US/Pacific
[...]
When did anyone check which of the OS replacement functions are
really necessary anymore? Some of them seem to be written to
replace really old and buggy implementations that may have gone
out of use by now anyway.

In some cases, they may never be needed again, but how can we know?

Can we remove those files that aren't used at all?

Yes, I suppose so. Since they'll live on in the CVS "attic," removing a source file is not irrevocable.

-Greg

Greg Ward wrote:

Thanks for doing all this work on the Windows port -- great progress!
I'm a little confused as to the benefit(s) of separating the objects
into so many libraries. Is this just to help reduce memory
requirements for dynamic linking? On Unix at least, it complicates the
builds considerably, as you need to know the correct link order for the
libraries or you get undefined symbol errors at load time.

The library splitting has several reasons:
First, it made it possible to circumvent the existing name
conflicts within the code, so that we now can compile with SCons
on Windows. Once we have those conflicts resolved in other ways,
that will reduce this particular benefit, of course.

Apart from that, I also used this as a tool to understand the
internal dependencies within Radiance better. It helped me to build
a mental map of the code. I don't think that the sequence of
libraries is a real problem. We just need to get that right for
each program once. As long as we use static libraries, all errors
will get caught at compile time anyway.

The current number of seperate libraries clearly isn't final. As
all the SCons stuff, I still consider this experimental, and will
continue to look for better ways to do things. We can't really
find the best build method without trying a few.

When did anyone check which of the OS replacement functions are
really necessary anymore? Some of them seem to be written to
replace really old and buggy implementations that may have gone
out of use by now anyway.

In some cases, they may never be needed again, but how can we know?

Since we always can reanimate files again at any later point (as
you note below), I think it's better to unclutter the tree
whenever we have a chance.

My question was targeted at figuring out which of those modules
are necessary on which platforms *right now*. The current
selection in makeall appears to include some random elements.
For example, getpagesize.c is included on at least one platform
without bmalloc.c, although the latter is the only file that
contains any reference to the former. I think it would be helpful
to go through this stuff with a finetoothed comb, but I lack
access to most of the platforms necessary to do this...

Can we remove those files that aren't used at all?

Yes, I suppose so. Since they'll live on in the CVS "attic," removing
a source file is not irrevocable.

Ok, let's do that then.

-schorsch

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/