Turn the RCX so the IR window points up.
Pull the right side of the battery compartment away from the top case. The left side will be held in place by the battery clip.
Lift the bottom of the black IR window. Pull it out.