Well yeah, both Kaido and Doflamingo cared a lot about their highest rank subordinates which were their actual crew mates. On other hand they didn't gave a shit about their lowest rank minions because they were easy to replace soooooo yeah, to some degree they were still horrible assholes. I would argue about Holy Knights NOT being weak though. I mean sure, at some point they deffinetly were really powerful but I'm quite sure that after Imu gifted them with immortality (well, at least our unwanted guests from Elbaf - Shamrock) their battle abilities could weaken. You know, because thanks to immortality they didn't have to depend on their stranght to survive. Look at example of Gunko: technically her body is soo easy to damage that even Usopps Grass Bomb was able to destroy upper half of her body while regular people don't get hurt THIS badly by them. She recovered but what would happen if she faced skilled conqueror haki user instead and would just take the hit without thinking? That's right, she would die immedietly. What's my point here? Well, I just think that ironically immortality made the Knights weaker because while they are still skilled fighters they lost their battle instincts which in normal setting would put their minds into "survival mode" during fights. Now despit fact that they have to finish their mission and get rid of every obstacle or else they will have to face Imus rage they can only run away. I assume it's because they kinda forgot how to fight to win AND survive. They dosen't dodge, they rarely block and it's all because they think immortality will always save them. Both Sommers and Killingham literally let Zoro and Sanji learn how to weaponize conqueror haki because they were taking all their hits. So while Holy Knights aren't full blown weaklings sense of security that immortality gave them deffinetly made them weaker than they should/could be.
Also about Franky well, I also don't remember when he said it but I'm about to relearn. Currently I'm watching Egghead for the first time because I got fed up of watching week to week and went on break. The point is, I will check if Franky did said that during Egghead arc. Although I feel like he could say that more recently, mayby even during Elbaf arc.