"Correct" Magic Number for ".hdr" Files

radiance-dev
1

Hi,

What is the correct magic number at the start of “.hdr” files? Radiance itself seems to write #?RADIANCE, but a very common one written by other software is #?RGBE. It is also attested that some files have none at all, though I haven’t seen this personally.

In “src/common/header.c:40”, #? is described as the “information header magic number”. Does this mean that the format is #?<anystring>? Or is it supposed to be #?RADIANCE, but using #? instead is a concession to #?RGBE existing “in the wild”

Might there be a historical angle here—like was #?RGBE part of an earlier version? Perhaps “.pic”, of which we’re told “The .pic extension is officially dead and buried”? (However, “doc/pdf/filefmts.pdf” currently lists “.pic” (pg. 28), and gives the magic number as #?RADIANCE and doesn’t mention “.hdr” at all).

(Bonus question: how about comments in “.hdr” headers? “doc/pdf/filefmts.pdf” (pg. 3) says they can be in “.rad”, but I’ve also seen them in “.hdr”. It’s not clear from the source whether this is encouraged.)

Thanks, -A

2

Hi Agatha,

It’s a good question, and you probably won’t like my answer.

Historically, there was no definitive magic number to any Radiance files, and the binary portion would follow whatever text appeared at the beginning up to an empty line that ended the header, i.e., “\n\n”. The length of a Radiance header has no limit. The header portion is traditionally used to contain the commands that created the file, metadata like the image exposure and a FORMAT string (described below).

The “#?” string was prepended at some point to offer assurance to my file readers that a file was really for them. It is used by all of the Radiance portable binary file types in the “#?RADIANCE” incarnation, though the “RADIANCE” string itself is ignored. Image files these days are expected to contain one of the following in their information header, usually at the end:

FORMAT=32-bit_rle_rgbe

or

FORMAT=32-bit_rle_xyze

The latter indicates XYZ rather than an RGB color space. Not all writers adhere to this specification, so there is some leeway in most of the HDR image readers. Following the “\n\n” string at the end of the header, there is a line of the form:

[-+][XY] vres [-+][YX] hres

to indicate the image dimensions and orientation. This is often the most reliable way to tell if a Radiance HDR image follows, though this kind of resolution string is also part of some other binary and even ASCII files in Radiance, but I won’t go into that. The usual English scanline ordering for a 512 wide by 300 high image would be:

-Y 300 +X 512

A single linefeed follows this string, which can have any number of spaces between the letters and first digits in the two dimensions. In all, a minimal header with resolution string for a calibrated image looks like this:

#?(anything)
EXPOSURE= (float value)
FORMAT=32-bit_rle_rgbe

-Y (yres) +X (xres)
(RGBE image data in scanline order)

I hope this helps!
-Greg

P.S. Unlike HDR images, some Radiance binary files, such as octrees and ambient caches, contain an additional magic number as the first (usually) two bytes following the header. This is our way of being confident we have the correct file type, since the “#?” string at the beginning is really generic to all Radiance files.

3

A few clarifying questions:

  1. Are comments (“#⋯”) allowed in “.hdr” files?

  2. If yes, is the intention maybe that “#?⋯” does double duty as (A) a comment (ignored by parsers that don’t understand it) and (B) as a magic number (when present, reassures a newer parser it’s on the right track)?

  3. Is it acceptable (but presumably discouraged) for a modern file to not have “#?⋯” at all?

  4. “doc/pdf/filefmts.pdf” says that “[a]t most one FORMAT line is allowed” (pg. 29). This seems to suggest that having zero format lines is allowed. If zero, should one assume RGBE? XYZE? Fail? Something else?

  5. Just to clarify regarding the minimal example you gave (noting with caution that the forum split at the “\n\n”), you provided an “EXPOSURE=” tag. However, per “doc/pdf/filefmts.pdf” (pg. 29) I think this is optional (I assume default 1.0), right?

It’s a good question, and you probably won’t like my answer.

Not to worry! I’m just trying to understand the history and state of things. In case you’re wondering, this is all in service of trying to “do the right thing” in regards to parsing and validating “.hdr” files. I’m happy just to get authoritative responses—and from the ideal person, too; thanks!

4

Hi Agatha,

Thanks for clarifying your needs regarding this inquiry – there are many non-standard HDR image writers out there. That is partly my fault, as (a) I don’t tend to think about file format adoption outside the Radiance community, and (b) this was one of the first formats I developed, so hadn’t considered all the ramifications.

  1. The ‘#’ character is not needed to delineate a comment – literally anything is allowed in the information header, and you can consider every line a type of comment. It’s just that some comments have meaning to some readers, like the “EXPOSURE=” line and other “VAR=” lines, including “FORMAT=”. (Incidentally, most readers should allow white space after the ‘=’ in such lines, but not before.) The information header continues until “\n\n” is encountered.

  2. The “#?” at the beginning was added for “file sniffers” that try to figure out a file format without understanding it, like the Unix file program. It has been partially successful in that regard, though the file program can misidentify other Radiance files as HDR images. It is also compatible with files that begin comments with ‘#’ as you point out, and not really definitive – the resolution string is a better indicator you have an HDR file, but still not 100% reliable.

  3. My readers don’t require files to start with “#?”, just in case some really old images are encountered. Newer readers should require it, and certainly any writer should write it (preferably as “#?RADIANCE”).

  4. If it were entirely up to me, I would make the “FORMAT=32-bit_rle_???e” string the main requirement, but since it follows a header of indeterminant length and comes in two versions, it’s not very sniffer-friendly. Also, there are HDR files out there that fail to include it, so it’s not 100% reliable, either.

  5. That is correct. The “EXPOSURE=” line is optional, 1.0 being the assumed value. Also, multiple values should be multiplied together – don’t just take the last one. So, my minimal example isn’t truly minimal in that sense.

My somewhat cavalier approach to reading Radiance HDR files is to accept whatever is in the header as valid, then look for the resolution string immediately following. If that’s there, then I proceed to the decoding stage, failing at that point if something is amiss. Worst case, we get garbage out.

-Greg