Whoever Dadan was, he would have to be an old friend of Garp, and would also have to be a skilled fighter. Dadan's first job would be to protect the kids. If anyone discovered the identity of their fathers, Dadan would have had to be the first line of defense. He would have to be strong. Probably vice-admiral level or better. Later, he would train them when they are old enough, but at first he was just their protector. I have a hard time believing that Garp would pick a non-marine for the job, as he wanted to train the kids to be strong marines.
I was all on the Gorosei bandwagon, but now the suggestion that it was Fleet Admiral Kong brought a smile to my face. Oda even hid Kong's face, so we know Kong will show up in future chapters sometime.
As for someone recognizing him, Raleigh has lived next door to fleet headquarters for 20 years and has managed to avoid attention all that time. He even kept the same name! If Kong is actually Dadan, he changed his name and went into obscurity in East Blue, where the marine presence is minimal and it would be fairly easy to avoid much attention. Your typical East Blue marines are pretty low on the totem pole so wouldn't be hobnobbing with the Admiralty. This means they would be unlikely to recognize Fleet Admiral Kong. He would likely just look like an old man with a couple of kids in a obscure little village.