It’s not that we don’t know which reference point is absolute, but there are still absolutely defined ‘points in space’, it’s that there is no absolute reference point, and so there are just ‘points in space’ relative to whatever arbitrary body you decide to make your reference frame.