maybe the parts of the vivre card point to the whole vivre card and not the person. so maybe if the rayleigh would leave the SA with his vivre card some people would know and try to bring him back.
maybe SA is his prison, like a retirement place and the marine HQ is controlling if he stays there...