Radiance Render Server

OK, so I've got this idea to make a Radiance Render server. Basically it will be a computer that sits in my closet, running Linux, that I can e-mail it a render job and have it e-mail back the finished images.

First off: Before I replicate a bunch of work, is there already something like this out there available? I know that Design Workshop has a render server available via a web form.

Second off: Would it even be feasible to use e-mail as the way to submit and receive jobs? The jobs would probably be rather large, so I guess I'll have to figure something out. Just seems very simple to make some programs that would take an e-mail and run with it than to have to build a web front end or something...

Thanks for any input,

Jeffrey McGrew

Jeffrey McGrew wrote:

OK, so I've got this idea to make a Radiance Render server. Basically
it will be a computer that sits in my closet, running Linux, that I
can e-mail it a render job and have it e-mail back the finished images.

First off: Before I replicate a bunch of work, is there already
something like this out there available? I know that Design Workshop
has a render server available via a web form.

I usually try not to advertise here, but if you ask... :wink:

Rayfront has a server component, which makes it possible to
send jobs to other machines on the local network. At the
moment, the user needs to control that manually, but I'm
working on queueing and load distribution functionality.

Second off: Would it even be feasible to use e-mail as the way to
submit and receive jobs? The jobs would probably be rather large, so I
guess I'll have to figure something out. Just seems very simple to
make some programs that would take an e-mail and run with it than to
have to build a web front end or something...

Client and server need access to the same disk space,
I don't think that using e-mail scales well enough.

If the server(farm) was offsite (which Rayfront doesn't
allow for reasons of security), other transport mechanisms
would become necessary. Actually, a simple scp might
do the trick.

-schorsch

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/

Jeffrey McGrew wrote:

First off: Before I replicate a bunch of work, is there already something like this out there available? I know that Design Workshop has a render server available via a web form.

The DW form is the only thing I've seen publicly too.

Second off: Would it even be feasible to use e-mail as the way to submit and receive jobs? The jobs would probably be rather large, so I guess I'll have to figure something out.

Well, it depends on what you need back. The really large files tend to be the octrees, the ambient cache files, and the Radiance pics, which all get generated from your rad files which are relatively small. Could you make do with jpeg versions of the images? They'd be much smaller than the 32 bit Radiance images. You could create a script that would take your uploaded .rad files (and maybe a rif file), runs the simulation, converts the pics to jpeg, and emails 'em back to you (or generates some html so that you could view the images online). All the bulk of the simulation would still be on the RRS (Radiance Render Server). Maybe the script could pack up the simulation as a tar.gz archive and a link to download the file could be added to the image html page; this would get around any email file size limits too.

Notice how there are no practical code samples here. I'm good with the ideas, but not with the programming. =8-) Keep us posted on your progress though!

···

----

      Rob Guglielmetti

e. [email protected]
w. www.rumblestrip.org

Jeffrey McGrew wrote:

OK, so I've got this idea to make a Radiance Render server. Basically it will be a computer that sits in my closet, running Linux, that I can e-mail it a render job and have it e-mail back the finished images.

First off: Before I replicate a bunch of work, is there already something like this out there available? I know that Design Workshop has a render server available via a web form.

Second off: Would it even be feasible to use e-mail as the way to submit and receive jobs? The jobs would probably be rather large, so I guess I'll have to figure something out. Just seems very simple to make some programs that would take an e-mail and run with it than to have to build a web front end or something...

Thanks for any input,

Jeffrey McGrew

_______________________________________________
Radiance-general mailing list
[email protected]
http://www.radiance-online.org/mailman/listinfo/radiance-general

if you want my arogant two-cent thoughts after some years of IT life: the only reason for doing it by email is what the name of your domain says: becausewecan.org

   1. You'll be moving a couple of dozen megabytes for any serious work,
      email gateways typically block very large emails, not to mention
      folks with slow links. A sysadmin may rip out vital parts out of
      someone using the system mail spool area as intermediate
      octree/image storage. ftp/http/scp is just better by design for
      large volume
   2. the server has to set up everything the specific job needs
      (version of binaries, environment variables, libraries, ...),
      which needs a bit of care with any method
   3. web-based leaps to mind before email does (the latter probably
      never leaps anyway...). Marc Fontoynont's group sat up something
      for their Genelux program long ago (sorry, don't have any further
      references to it)
   4. how do your users pack their jobs ? zip,tar ? How do they verify
      they have everything included ?
   5. why setting up a server anyway? For an occasoanl user the various
      boot-radiance-from-cd concepts are probably much quicker (right on
      their machine), for bulk processing a render farm with file
      sharing is approx 3 orders of magnitude faster and easier to manage
   6. for public usage the server would have to implement some resource
      management, otherwise any job may block the resources for an
      unpredictable time

if you still want to go and dig in (good!), my personal recommendation would be PHP scripts on a webserver (very preferably Apache). Actually I thought about it years ago at ISE, but just hadn't found a good reason to offer it publicly. Internally we used our own job distribution system (based on NFS) at ISE.

cheers
Peter

···

--
pab-opto, Freiburg, Germany, http://www.pab-opto.de
[see web page to check digital email signature]

Peter Apian-Bennewitz wrote:

Jeffrey McGrew wrote:

>
> OK, so I've got this idea to make a Radiance Render server.

   3. web-based leaps to mind before email does (the latter probably
      never leaps anyway...).

Dont ask me why I didn't mention this an hour ago.

Probably because I'm still more thinking of it as a project
than a product. But I've been working on a web based online
version of Rayfront for quite a while now. It's in beta, and
the payment infrastructure is currently being integrated.

The people who commissioned this will offer it as a service,
or sell it as a "render appliance". You can buy a box to be
installed in a rack in your server room, and all you need
to access it for running simulations is a web browser with
the flash plugin.

   4. how do your users pack their jobs ? zip,tar ? How do they verify
      they have everything included ?

Upload a DXF file, compressed with zip or gzip if it
gets too large otherwise. Other types of upload will be
easy to add if there's a real need.

   5. why setting up a server anyway? For an occasoanl user the various
      boot-radiance-from-cd concepts are probably much quicker (right on
      their machine), for bulk processing a render farm with file
      sharing is approx 3 orders of magnitude faster and easier to manage

How about a render farm in behind a server based interface?
Those boot-radiance-from-cd concepts also don't come with
tools for material assignment and simulation control

   6. for public usage the server would have to implement some resource
      management, otherwise any job may block the resources for an
      unpredictable time

The available CPUs need to be adequate for the expecded
load. In an in-house situation, you just file jobs into a
queue. On a commercial service you can buy "render points",
some of which get eaten up by each simulation you run.
Other than that you have user and group management, so that
several people from the same organisation can run their jobs
and it all gets billed together.

if you still want to go and dig in (good!), my personal recommendation
would be PHP scripts on a webserver (very preferably Apache).

Buzzword alarm:
The online version of Rayfront is a multi-tier solution.
It uses the Python-based web application server Zope running
behind Apache, with XML-RPC connections to the Flash client and
to the simulation backend.

-schorsch

···

--
Georg Mischler -- simulations developer -- schorsch at schorsch com
+schorsch.com+ -- lighting design tools -- http://www.schorsch.com/

if you want my arogant two-cent thoughts after some years of IT life: the only reason for doing it by email is what the name of your domain says: becausewecan.org

I love the input, that's why I posted it out to the list! :wink:

  1. You'll be moving a couple of dozen megabytes for any serious work,
     email gateways typically block very large emails, not to mention
     folks with slow links. A sysadmin may rip out vital parts out of
     someone using the system mail spool area as intermediate
     octree/image storage. ftp/http/scp is just better by design for
     large volume

  2. the server has to set up everything the specific job needs
     (version of binaries, environment variables, libraries, ...),
     which needs a bit of care with any method

.tar file is sent to the 'RRS' (thanks for that) and then it's unpacked, and any user-defined .cal's and such will be included within that archive. The basic assumption is that everyone using this (which would be my company, which right now consists of me :wink: ) would be on the same page, Radiance-wise.

  3. web-based leaps to mind before email does (the latter probably
     never leaps anyway...). Marc Fontoynont's group sat up something
     for their Genelux program long ago (sorry, don't have any further
     references to it)

Web stuff and FTP are all well and good, and a great way to go, if I want to pay extra for a static IP and the task of managing my own server. I'm mobile, working from a laptop, and while I can run stuff under Cygwin (and do) it would be cooler/better/more feasible to have the RRS do this while I'm working on something else and/or in transit.

  4. how do your users pack their jobs ? zip,tar ? How do they verify
     they have everything included ?

I've got a flowchart that I worked out, that makes it e-mail back error messages if something is left out or if there is an error.

  5. why setting up a server anyway? For an occasoanl user the various
     boot-radiance-from-cd concepts are probably much quicker (right on
     their machine), for bulk processing a render farm with file
     sharing is approx 3 orders of magnitude faster and easier to manage

Two things: When rendering, I would like to work on my computer for other things too, and those other things require Windows. Currently I run Cygwin on my laptop, and use Radiance within there. This is working well, but I would love to be able to hand these jobs off to a faster computer, and a desktop/server gives a lot more bang for the buck than a laptop in raw processor speed. It would also allow for the RRS to run whatever OS is best on whatever hardware I can afford, and then I can move the RRS to whatever platform I need, or in the future to a cluster...

Second is that it would be cool. No, Seriously. I work in Revit, dump my models to AutoCAD, tweak layers and such, export to Radiance, send the job off, and then keep working in Revit while my job 'bakes' on the RRS and is sent back to me when done. How cool would that be? My little RRS working away, day and night... Makes animations much more feasible too!

  6. for public usage the server would have to implement some resource
     management, otherwise any job may block the resources for an
     unpredictable time

This would primarily be for my own use, so I'm gonna start out small.

if you still want to go and dig in (good!), my personal recommendation would be PHP scripts on a webserver (very preferably Apache).

Yeah, if I go with a web-thing it would probably be Apache on Mandrake, using Python (just because it's what I'm a little familiar with) or something.

Thanks again, keep it coming!

Jeffrey McGrew

···

A: See No. 3 below
A: I was thinking that this would be a part of the solution, a .zip or