I don’t think Moria is going to cause any conflict of loyalties for Yamato, the Kozuki siblings, or the Scabbards. They owe Luffy and the Strawhats way too much.
If anything, Moria (if he joins CG) will be the one facing conflict of loyalties. And even then it won’t be much of a conflict...