No, no, no...
this sounds quite good. You're not reading too far into things. Now that you mention it, it sounds logical. Remember Kon? Toki first meets him in the virtual Edo History game (he still has brownish-blond hair there). Then, when he enters Amatsuki and meets Kon there (now: black hair) he says that he has just meet him but Kon assures him that this meeting is from 2 years ago. I first thought it was some kind of time slip but with your theory... it sounds correct, too.