So Roger was also father of Shanks and Shamrock? Garling uses it to lure him out and he was able to recover only one of them? Is that what Garling found there?
I bet it is exactly that. They know that going forward will destroy all the tower and its inhabitants. That is why they all stopped V and called it a monster, because he wanted to keep climbing.
I think Shanks was considered (by Roger and crew) the next joy boy, and even himself was convincend of that. Then when luffy ate the fruit, he understand that his role is a different one, to make joy boy happen through luffy. He is now departing because his duty is fulfilled and he can return to...