I think Sarith meant to say that the “up” vector can be any direction but parallel to the surface normal (i.e., perpendicular to the surface). The up vector is used to orient the azimuth of the hemisphere’s coordinate system. If the up vector is parallel to the “zenith” direction of the hemisphere you are trying to orient, then this is undefined. It is exactly the same as specifying a view up direction vector (-vu) that is parallel to the view direction (-vd) in rpict.
By convention, the up vector is defines the Y-axis orientation. The X-axis then points to the right in a right-handed coordinate system when viewed from the front side of the surface, looking down the Z-axis of the hemisphere. All coordinate systems in rfluxmtx can be specified as left-handed by prefixing their name with a minus sign (’-’). This is necessary to replicate Klems coordinate symmetry and the like.