You are, in a very real way, over thinking it. He is invisible, it is completely irrelevant because he hasn't attacked me. If he goes invisible, and I was witness to it, I would obviously be able to put up defensive nen in preperation. But if I suddenly get punched in the face, I would be able...