Kidd's backstory is pretty great! Going to take a shot in the dark and say Victoria died because she confronted the gang leader for laughing at her dream. Kidd's reaction to killing anyone who laughs at his dream would make a ton of sense if that were the case.
Like how the concept of freedom is prevelant in the "good" Yonkou crews. Shanks has his allies and a large crew, but, like Luffy, he doesn't force them to sail with him, ala Kaido and Big Mom. They are free to live their own pirates lives and likely only join in when Shanks requests their help or, like Whitebeard, they come whenever they feel they are needed.