It seems a recurring theme for the top swordsmen to seemingly not use Devil fruits. Rayleigh doesn't seem to use one, Shanks & Zoro definitely don't, neither do Vista or Shiryuu (from what we've seen of them at least). I have always seen swordsmen as characters that live and die by their blade, and their faith in said blades. Kaku has been the only one in canon I can remember that used a fruit and Zoro ended up defeating him (there was of course that Shiki guy, but movie characters should not be considered as canon). Mr.1 is a bit ambiguous, as he himself does not consider himself a swordsman while many others do. I think it'd be pretty cool for the best swordsmen to rely on the will of their sword, obviously developing Haki is a must and I'd imagine is perhaps a logical progression of their skill. Mihawk can probably hit Logias if Vista can, meaning Zoro may awaken the ability also. I think if he sees Luffy hurt a Logia with regular attacks then he'll train to do the same.
Ooooook.
Rayleigh, Shanks, Vista and Shiryuu. Please show me where they DON'T use a devil fruit. I'm gonna go by the same logic you're going by here. You haven't seen a DF so it doesn't exist, I haven't seen a DF so it must exist.
Point is: don't go around crying when Rayleigh and Shanks show off their fruits like people were when Whitebeard showed off his. All the "OMG how lame WB has a DF" complaints died after two chapters of earthquakes.
I highlighted "from what we've seen of them" for obvious reason. We've seen nothing of them. Of the four you mentioned there, we've seen a lot of Shanks, but I've never seen him fight. Of the four, the one who has had the most on screen fight time is Rayleigh, and all I saw were three sword attacks and one haki blast.
I'm still not getting all these attempts at making a "class" in One Piece that should be free of Devil Fruits other than attempting to compare them directly to Zoro. Wouldn't you be happier if everyone in the world had a DF and Zoro didn't? And this belief seems to go hand in hand with old famous guys.
Make a list of new famous guys, and they all (almost) have devil fruits. Why should all the old famous guys have ignored devil fruits?