There is a phenomenon in which, if you isolate members of some species, they tend to either evolue to be bigger or smaller. That's what happened to the Komodo dragon.
But, in One Piece, only the strong guys get real big. You don't see normal citizens getting enormous. Just like irl, you rarely...