Hi all,
I'm trying to read the binary output from dctimestep run with the -od
argument. The idea is that the binary files appear to be a lot faster to
save and load than text. However, I'm having a problem reading the binary
values.
Values less than 512 read in just fine. However, binary values greater than
512 are being converted to what look like random values between 31 and 32.
Upon looking at the binary file, it looks like the first three bytes of
each double are zero. Of course, I'm not even sure of the endianness of the
data.
I'm also not sure how to test whether the issue is in my reader or in the
dctimestep output. Here's my C# code to read the values, in case anyone
wants to do a deep dive here:
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(buffer, 0, ncols * ncomp
* sizeof(double));
for (int j = 0; j < ncols; j++)
{
double r = BitConverter.ToDouble(bytes, j * ncomp * sizeof(double));
double g = BitConverter.ToDouble(bytes, (j * ncomp + 1) *
sizeof(double));
double b = BitConverter.ToDouble(bytes, (j * ncomp + 2) *
sizeof(double));
irradiance.Add(Bright(r, g, b), path);
}
Any help debugging this would be appreciated!
Nathaniel
Hi Nathaniel,
The binary output of dctimestep is just a dump of float (or double) values in memory, and can be read into a float (or double) array using fread(), or the Radiance equivalent, getbinary().
I don't know what System.Text.Encoding.ASCII.GetBytes() does, but it's probably the wrong thing. You need to make sure you are reading the file in a binary mode, otherwise Windows likes to convert "\r\n" sequences to "\n", which will mess up your input.
I also don't know how the BitConverter class works, but you need to be sure it isn't swapping bytes.
I would recommend instead calling:
_setmode(_fileno(fp),_O_BINARY);
while (fread(rgb, sizeof(double), 3, fp) == 3) {
process(rgb);
}
Cheers,
-Greg
···
From: Nathaniel Jones <[email protected]>
Date: April 16, 2018 5:42:06 PM PDT
Hi all,
I'm trying to read the binary output from dctimestep run with the -od argument. The idea is that the binary files appear to be a lot faster to save and load than text. However, I'm having a problem reading the binary values.
Values less than 512 read in just fine. However, binary values greater than 512 are being converted to what look like random values between 31 and 32. Upon looking at the binary file, it looks like the first three bytes of each double are zero. Of course, I'm not even sure of the endianness of the data.
I'm also not sure how to test whether the issue is in my reader or in the dctimestep output. Here's my C# code to read the values, in case anyone wants to do a deep dive here:
byte bytes = System.Text.Encoding.ASCII.GetBytes(buffer, 0, ncols * ncomp * sizeof(double));
for (int j = 0; j < ncols; j++)
{
double r = BitConverter.ToDouble(bytes, j * ncomp * sizeof(double));
double g = BitConverter.ToDouble(bytes, (j * ncomp + 1) * sizeof(double));
double b = BitConverter.ToDouble(bytes, (j * ncomp + 2) * sizeof(double));
irradiance.Add(Bright(r, g, b), path);
}
Any help debugging this would be appreciated!
Nathaniel
Nathaniel, the ReadSingle() or ReadDouble() methods of the System.IO
BinaryReader class may do your job. Give those a try. If they don't work,
try ReadBytes() in conjunction with those BitConverter methods you've been
working with. Key, though: you probably want to be using BinaryReader.
···
--
Randolph M. Fritz || [email protected]
On Mon, Apr 16, 2018 at 5:42 PM, Nathaniel Jones <[email protected]> wrote:
Hi all,
I'm trying to read the binary output from dctimestep run with the -od
argument. The idea is that the binary files appear to be a lot faster to
save and load than text. However, I'm having a problem reading the binary
values.
Values less than 512 read in just fine. However, binary values greater
than 512 are being converted to what look like random values between 31 and
32. Upon looking at the binary file, it looks like the first three bytes of
each double are zero. Of course, I'm not even sure of the endianness of the
data.
I'm also not sure how to test whether the issue is in my reader or in the
dctimestep output. Here's my C# code to read the values, in case anyone
wants to do a deep dive here:
byte bytes = System.Text.Encoding.ASCII.GetBytes(buffer, 0, ncols *
ncomp * sizeof(double));
for (int j = 0; j < ncols; j++)
{
double r = BitConverter.ToDouble(bytes, j * ncomp * sizeof(double));
double g = BitConverter.ToDouble(bytes, (j * ncomp + 1) *
sizeof(double));
double b = BitConverter.ToDouble(bytes, (j * ncomp + 2) *
sizeof(double));
irradiance.Add(Bright(r, g, b), path);
}
Any help debugging this would be appreciated!
Nathaniel
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
Randolph's solution worked here. The tricky part was to be reading the file
header as strings and then the binary data from the same stream. I ended up
creating a BinaryReader when a format other than ascii is specified and
initializing it with the BaseStream from my StreamReader so as not to have
to open the same file twice. This requires tracking the current position in
the stream, so it's not completely clean, but still simpler than what I had
before. It's about three times faster than reading the text input.
Nathaniel
···
On Mon, Apr 16, 2018 at 9:48 PM, Randolph M. Fritz <[email protected]> wrote:
Nathaniel, the ReadSingle() or ReadDouble() methods of the System.IO
BinaryReader class may do your job. Give those a try. If they don't work,
try ReadBytes() in conjunction with those BitConverter methods you've been
working with. Key, though: you probably want to be using BinaryReader.
--
Randolph M. Fritz || [email protected]
On Mon, Apr 16, 2018 at 5:42 PM, Nathaniel Jones <[email protected]> > wrote:
Hi all,
I'm trying to read the binary output from dctimestep run with the -od
argument. The idea is that the binary files appear to be a lot faster to
save and load than text. However, I'm having a problem reading the binary
values.
Values less than 512 read in just fine. However, binary values greater
than 512 are being converted to what look like random values between 31 and
32. Upon looking at the binary file, it looks like the first three bytes of
each double are zero. Of course, I'm not even sure of the endianness of the
data.
I'm also not sure how to test whether the issue is in my reader or in the
dctimestep output. Here's my C# code to read the values, in case anyone
wants to do a deep dive here:
byte bytes = System.Text.Encoding.ASCII.GetBytes(buffer, 0, ncols *
ncomp * sizeof(double));
for (int j = 0; j < ncols; j++)
{
double r = BitConverter.ToDouble(bytes, j * ncomp * sizeof(double));
double g = BitConverter.ToDouble(bytes, (j * ncomp + 1) *
sizeof(double));
double b = BitConverter.ToDouble(bytes, (j * ncomp + 2) *
sizeof(double));
irradiance.Add(Bright(r, g, b), path);
}
Any help debugging this would be appreciated!
Nathaniel
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
Glad that worked. You should be OK reading the header as binary data, though. All the programs that write headers use binary mode, so you just get back what was put in. You don't actually want to do line-end conversion for Radiance headers.
Cheers,
-Greg
···
From: Nathaniel Jones <[email protected]>
Date: April 17, 2018 3:02:19 PM PDT
Randolph's solution worked here. The tricky part was to be reading the file header as strings and then the binary data from the same stream. I ended up creating a BinaryReader when a format other than ascii is specified and initializing it with the BaseStream from my StreamReader so as not to have to open the same file twice. This requires tracking the current position in the stream, so it's not completely clean, but still simpler than what I had before. It's about three times faster than reading the text input.
Nathaniel
On Mon, Apr 16, 2018 at 9:48 PM, Randolph M. Fritz <[email protected]> wrote:
Nathaniel, the ReadSingle() or ReadDouble() methods of the System.IO BinaryReader class may do your job. Give those a try. If they don't work, try ReadBytes() in conjunction with those BitConverter methods you've been working with. Key, though: you probably want to be using BinaryReader.
--
Randolph M. Fritz || [email protected]
On Mon, Apr 16, 2018 at 5:42 PM, Nathaniel Jones <[email protected]> wrote:
Hi all,
I'm trying to read the binary output from dctimestep run with the -od argument. The idea is that the binary files appear to be a lot faster to save and load than text. However, I'm having a problem reading the binary values.
Values less than 512 read in just fine. However, binary values greater than 512 are being converted to what look like random values between 31 and 32. Upon looking at the binary file, it looks like the first three bytes of each double are zero. Of course, I'm not even sure of the endianness of the data.
I'm also not sure how to test whether the issue is in my reader or in the dctimestep output. Here's my C# code to read the values, in case anyone wants to do a deep dive here:
byte bytes = System.Text.Encoding.ASCII.GetBytes(buffer, 0, ncols * ncomp * sizeof(double));
for (int j = 0; j < ncols; j++)
{
double r = BitConverter.ToDouble(bytes, j * ncomp * sizeof(double));
double g = BitConverter.ToDouble(bytes, (j * ncomp + 1) * sizeof(double));
double b = BitConverter.ToDouble(bytes, (j * ncomp + 2) * sizeof(double));
irradiance.Add(Bright(r, g, b), path);
}
Any help debugging this would be appreciated!
Nathaniel
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
_______________________________________________
Radiance-dev mailing list
[email protected]
https://www.radiance-online.org/mailman/listinfo/radiance-dev
@Nathaniel_Jones - would you mind posting your solution here? I am trying to do the same thing but must be missing something…
1 Like
@Timur_Dogan - This was from so long ago, I didn’t recognize it at first. I haven’t used C# in a while because I wasn’t satisfied with the speed I could get from it.
That said, if you want to read radiance files (ascii or binary) with C#, here is a strategy:
StreamReader file = new StreamReader(pathToFile, System.Text.Encoding.ASCII);
// ...
// Read the header as text.
// Important: Count the number of characters that you read (including newline characters).
// ...
switch (format) {
case Format.ASCII:
while ((line = file.ReadLine()) != null && !string.IsNullOrEmpty(line)) {
string[] entries = line.Split(null);
for (int i = 0; i < entries.Length / 3; i++) {
double r = double.Parse(entries[3 * i]);
double g = double.Parse(entries[3 * i + 1]);
double b = double.Parse(entries[3 * i + 2]);
irradiance.Add(Bright(r, g, b));
}
} break;
case Format.FLOAT:
file.BaseStream.Seek(charCount, SeekOrigin.Begin);
BinaryReader bin = new BinaryReader(file.BaseStream, System.Text.Encoding.ASCII, true);
while (bin.PeekChar() != -1) {
for (int j = 0; j < ncols; j++) {
double r = bin.ReadSingle();
double g = bin.ReadSingle();
double b = bin.ReadSingle();
irradiance.Add(Bright(r, g, b));
}
} break;
case Format.DOUBLE: // Same with bin.ReadDouble()
}
This is part of a much larger package I was working on back then targeting Grasshopper. I have since rewritten the whole thing with Numpy, which blows Grasshopper out of the water in terms of speed and data size limitations.
Nathaniel
1 Like
Hi @Nathaniel_Jones, Thank you for sharing this. I have a follow up question. Does numpy handle Radiance binary outputs smoothly? I have been looking to pandas and also some data formats like Parquet to make Radiance matrices manageable/query friendly but if it can be done with numpy then I prefer using the smaller library/dependency here.
@Mostapha, it’s as simple as
import numpy
with open(path, 'rb') as reader:
# ... read the header, and then
data = numpy.fromfile(reader, dtype=numpy.float32).reshape(nrows, ncols, ncomp)
Using numpy you can duplicate most of the functionality of dctimestep using the BLAS library, which saves a lot of time when working with large matrices. I also have some Python methods that handle the various curve balls that can come up (e.g. missing NROWS or NCOMPS from the header).
2 Likes
This is great. I struggled with matrix ops on the old OpenStudio stuff and I’m struggling with it now. When you want to work with this data in typical viz tools, it’s helpful to work on the stuff directly.