Pcomb read error

OK I’m ready to kill.

Working on an ass-simple script, in Python, to process some images. Trying to send something like this:

pcomb -e “lo=if(li(1),1,0)” tmp.hdr > tmp2.hdr

Double quotes are needed because I’m on Windows, which I’m super happy about. When I pass this command exactly like the above to the cli from the script, I get:

tmp.hdr: read error

But when I LITERALLY copy and paste that exact string from the console (my script prints what it’s about to pass to the cli), it WORKS FINE.


and, hi everyone. =)

  • Rob

P.S. maybe this is more illustrative:

(stdout from the Python script):
Running command:
$ pcomb -e “lo=if(li(1),1,0)” tmp.hdr > tmp2.hdr
tmp.hdr: read error

(looking at file sizes, we see that pcomb had some issues):
ls -l tmp*.hdr
-rw-r–r-- 1 rguglielmetti 1049089 224334 Feb 21 21:14 tmp.hdr
-rw-r–r-- 1 rguglielmetti 1049089 623 Feb 21 21:14 tmp2.hdr

(running the exact command from the cli directly):
pcomb -e “lo=if(li(1),1,0)” tmp.hdr > tmp2.hdr

(seems to work (does work, I verified the output hdr is good)):
ls -l tmp*.hdr
-rw-r–r-- 1 rguglielmetti 1049089 224334 Feb 21 21:14 tmp.hdr
-rw-r–r-- 1 rguglielmetti 1049089 224527 Feb 21 21:14 tmp2.hdr

P.P.S. this is a simple os.system call.

WTF * infinity

Is it possible that your os.system call is somehow calling the PowerShell? I know nothing about Windows, but I’ve heard the PowerShell messes with binary i/o and should be avoided.

I can confirm that the error message is coming from pcomb and indicates a problem reading in the RGBE scanlines from the input file.

Try running the same command with a tiny picture, maybe 6x6 pixels or something. Just as a test.

1 Like

Hi Rob,

You might also try using subprocess for this rather than an os.system call, although I am not sure it will help given what Greg said.

import subprocess as sp
cmd = ['pcomb', '-e', 'lo=if(li(1),1,0)', 'tmp.hdr', '>', 'tmp2.hdr']
proc = sp.Popen(cmd, shell=True)


1 Like

Thanks guys! yeah, subprocess didn’t work either, in this case. I am using it successfully in another place in this horrible program, but I guess this is just an incompatibility here with pcomb. I saw a thread from a few years back where Greg helped another user and traced this nonsense to this PowerShell BS, but truthfully I have no idea what I’m using nor have I found a way to tell wtf I’m using. I ASSumed I was using the “standard” Windows shell since I haven’t installed Power Shell or any other dodads. I did learn tho that typing “bash” at this command line takes me to a semi-proper bash shell. So we can ASSume again that I am probably using some hopped up shell, but this is just what comes in the box with a stock Windows10 installation.

After spending hours and hours on this bs, I have moved on to the next piece of this port, and am hitting new issues. This is SO MUCH FUN.

I cannot believe this pOS (Windows) is still this popular. It’s as bad as ever.

hoooooboy, y’all ready for this? The issue is that os.system calls don’t wait for completion before moving onto the next line. I stg I’m ready to kill for real now.

The pcomb command was failing at runtime because the hdr image it was intending to operate on hadn’t completed writing yet!! This is why when I would try it at the cli it worked because the file was there by the time I’d copy/paste the next command in. So now I have sleeps installed here, so this gd thing can just run. This is, uh, lame.

What the hell is going on, people? I mean, how does anyone write python code to do some simple Radiance workflows these days? I also had tried Alstan’s subprocess.Popen suggestion as well as another permutation of that module (subprocess.run), and despite being sold as more up to date modules that will wait for one process to complete to start the next, they do not work in this case. Apparently I’m supposed to use subprocess.communicate now? This is unbelievably complicated and I’m really having a hard time believing this is the way everyone’s doing things on Windows these days. WTF am I doing wrong??

Hi @Rob_Guglielmetti,

The command appears to be working fine on my system, although I had to change the double quotation characters for python.

Your command (that works on my Windows CLI but not on Python 3.6):
pcomb -e “lo=if(li(1),1,0)” 15hrs.hdr > tmp2.hdr

The command that works on Windows and Python 3.6 (at least on my system):
pcomb -e "lo=if(li(1),1,0)" 15hrs.hdr > tmp2.hdr




1 Like

Hi Sarith,

Thanks for taking a look too. The stupid quotation characters are being
inserted without my consent when I copy from my text editor to Chrome on
Windows here.

The command works just fine, as long as the input hdr exists in a complete
state. The issue (which took me an embarrassingly long time to figure out)
is that the preceding command in the script–which creates the hdr–is not
finished writing the file in its entirety before this command runs. The
read error is happening because it’s trying to read an incomplete file.
When I put in a time.sleep(5) after the command to make the hdr, the script
runs fine. I noticed all this when I randomly had an Explorer window open
to the directory where everything was being written. The file size
eventually gets to half a megabyte or whatever, but when the script runs,
you can see it’s only like 20% written before the pcomb command executes,
and then of course the read error.

I am *gobsmacked *that this is happening.

This is also the first time in my entire life I’ve used the word

Hey Rob,

I agree that piping, and re-directing stdin and stdout are a bit clunky in Python. Which is why I still stick to BASH scripts. Here is an example with proc.communicate():

    with open(self.oct_file, 'w') as fh_oct:
        proc = Popen(cmd, shell=False, stdin=PIPE, stdout=fh_oct, stderr=PIPE,
        stdout, stderr = proc.communicate()
        exit_code = proc.returncode

Try to keep your sanity. It can all be done. Somehow.

1 Like

Hi Rob,

The quotes in Windows seem to cause a lot of undue confusion, in my opinion. So far as I can tell, there’s no need to use quotes at all when you’re in Windows, so long as your string doesn’t include whitespace. In the Unix world, you only need quotes in order to treat the $ operator as a string literal, which it doesn’t seem you need to do in your case.

I don’t think the quotes have anything to do with this particular issue. Again, the fancy quotes were just an artifact of Chrome/Windows, they are not in my actual code. I do feel like single quotes didn’t work for this implementation, but maybe the error I got when I tried that was more related to os.system on Windows. I dunno, that was like 24 straight hours ago.

Axel, that to me is an insane amount of housekeeping needed to execute a simple shell command. But it’s looking like that’s the way forward, sigh. And yeah, bash is great, but there is so much more available when you can access Python libraries e.g. honeybee-radiance! (Shameless plug sorry-not sorry)

Did you try asking the Honeybee guys? Seems they would have encountered this before.

What a saga!

I did, they use Popen a lot too, and have their own functions for their most commonly-used radiance commands. I need to tuck in and grapple with this crap I guess. The interscape is rife with deets on how horrendous this is though.

It comes down to the simple and remaining long-standing fact that Windows is some broken sh*t. We are talking about fundamental UNIX: the toolbox model, and Python has to jump through all kinds of hoops to emulate support for that. :expressionless:

1 Like

Just in case anyone’s interested, after much trial, error, and obvious cursing; after trying all (well, most of) the suggestions here; I finally got this to work (on Windows; it worked all along (as in, like, 48 HOURS AGO) with os.system() on *nix, but I digress).

cmd = f"rtpict -vf {view_name} {res_string} {img_opts} tmp.oct > tmp.hdr"
proc = subprocess.run(cmd.split(' '), shell=True, capture_output=True)

So, subprocess.run was added (Python 3.5) to roll up os.Popen and communicate and whatnot. And that capture_output=True parameter is what carried this sh!tshow to the finish line. I tried wait()'s and everyothergoddamnedthing(), and I have the commented code and the seething anger to prove it. This was the only way I could get what I thought was a simple request to act like it does from bash, sh, zsh, and even Python on any other OS but Windows–namely, wait for a shell command to completely finish before executing the next line in the script.

Since my only issue was this delayed write messing up the next command (a pcomb read), I’ve only changed this one call for now. But I will play with the rest later. I’m not sure if subprocess.run will eat a command with backticks from a split string list (or other unix toolbox exotica common to Radiance workflows), but we always have shlex and like thirty other modules that may or may not work, right? Sigh.


Thanks again to all who responded so quickly to my initial question!

– Rob

1 Like

I am sure you are done wanting to think about this, and I am sorry I didn’t see this sooner, but as I do not have easy access to a windows machine I didn’t want to give untested advice. But I think the discussion could hold general interest for anyone radiance scripting with python. It is also an opportunity for a shameless plug, and call for help. But first…

Your problems stemmed from using the the “>” redirect in the command, the “correct” way is to let python manage the file handling. Your ultimate solution:

subprocess.run(cmd.split(' '), shell=True, capture_output=True)subprocess.run(cmd.split(' '), shell=True, capture_output=True)

worked because of the 'shell=True parameter. While I don’t have Windows, passing shell=True to os.system likely works as well:

os.system(f"rtpict -vf {view_name} {res_string} {img_opts} tmp.oct > tmp.hdr", shell=True)

Now, you probably came across all sorts of advice on the internet about how this is VERY BAD! and causes all sorts of exploitable security issues in your code, but for a simple script there is nothing wrong with doing it this way, and in any case, your subprocess.run command exposes shell access in the same way.

For blocking commands, the correct subprocess command (to avoid the communicate() confusion) is subprocess .call():

from subprocess import call
import shlex #does shell style lexical analysis

cmd = shlex.split(f"rtpict -vf {view_name} {res_string} {img_opts} tmp.oct")

f = open("tmp.hdr", "wb")
call(cmd, stdout=f)

You are right that things get more complicated when you want to handle piping, this is where Popen becomes necessary. Say I wanted to run your command the old fashioned vwrays | rrtrace way:

from subprocess import Popen, PIPE
import shlex

vwrays = shlex.split(f"vwrays -ff -vf {view_name} {res_string}")
rtrace = shlex.split(f"rtrace  {res_string} -ld- -ffc {img_opts} tmp.oct")
out = open("tmp.hdr", "wb")
f = Popen(vwrays, stdout=PIPE)
Popen(rtrace, stdin=f.stdout, stdout=out).communicate()

This is all certainly way more complicated than a simple script should be, but these commands help navigate the cross platform differences to help avoid ugh… Windows headaches and such. Fortunately, others (like me) have spent a bunch of time coding the same type of thing over and over again, soo, now time for a shameless plug: I wrote a python library specifically for dealing with the types of tasks that radiance scripters encounter. The bulk of the library is to help write consistent and intuitive command line programs with powerful argument parsing using optional configuration files, but the work-horse method is pipeline():

from clasp import script_tools

vwrays = f"vwrays -ff -vf {view_name} {res_string}"
rtrace = f"rtrace {res_string} -ld- -ffc {img_opts} tmp.oct"
pcomb = "pcomb -e 'lo=if(li(1),1,0)' -"
cmds = [vwrays, rtrace, pcomb]

script_tools.pipeline(commands, outfile="tmp2.hdr", writemode="wb", close=True)

Isn’t it beautiful? I am not sure what you are doing with tmp2.hdr, but you can easily extend the pipe and get rid of all the intermediate files. You can even delegate the octree creation with some special syntax:

rtrace = f"rtrace {res_string} -ld- -ffc {img_opts} $(oconv {scene_files})"

And you may ask yourself: So how do get this wonderful tool?

pip install clasp

And you may ask yourself: where do you learn how to use it? right here

And you may ask yourself: Well, I heard your shameless plug, and this all seems wonderful, but I also recall you needed help?

Yeah, so now I must confess: clasp has not been tested on Windows and I do not expect it to work out of the box. But it is fully open source project and I would love a collaborator to help build up the test suite and debug on windows. If that is you, please get in touch.

Apologies for hijacking a windows thread to promote a tool untested on windows, and a big apology to David Byrne.

1 Like