I believe we can achieve a nice theory without taking some stuff for granted, some debatable ones. Don't hypothesize that much.
What do we know about him?
1) Erina is incredibly afraid of her father, to traumatized levels;
2) He's an elitist, segragationist dick;
3) He's so much of a shitty person that the AE judges think that he should have been banished from Tootsuki;
4) He had not been banished until now, even though...
5) He's not Erina's guardian;
6) Every trace of him had been erased from the academy;
7) By the time Hisako was 6~9 (personal guess of her age), he was already out of Tootsuki;
8) He started lecturing Erina by the time she was 6;
9) We have no indication of his age.
So, trying to explain each of those points. This post's going to be huge, just a warning:
1) What could have made Erina so afraid of her father? We're still talking about a shonen and he was not arrested, so no physical punish or anything like that. W know that from an early age she had shown signs of her amazing tasting skill. He was an already acomplished cook with a gold mine as a child, he could achieve perfection using her. That's where it probably started.
So, to start what IMO makes sense: since Erina began to talk and showed signs of her God Tongue, Azami has been making her somekind of a "slave-judge", making her taste every dish he creates. That makes Erina anxious, tired, stressed. Senzaemon gets pissed with his son. Not only that, but probably just after she turns able to handle knifes and etc. (here we're using info 8 from the above list), and seeing her unlimited potential, he starts to teach her. He is then even more demanding of her. Senzaemon get's more pissed.
Now, getting to 3~7) from above
We don't know when Erina's mother died. Based on 7), before Hisako was 6~9. So, by the time Erina starts to cook, her mother had already died. Did Azami have anything with that? Probably not.
Following what I've said at the previous paragraph up there, Azami's rigid training with Erina crosses the limit completely (I won't try to guess it). Senzaemon "banishes" Azami from Erina's life (protecting her from him even by erasing his traces from the mansion), but since he's his son and deeply talented, doesn't ban him from Tootsuki, since it would taint his career and a possibility of a 2nd chance (that's reasonable, more from it up next). That explains 5 and 6. That accomodates Erina knowing how to cook by the time Jou was the Nakiri's personal chef (she wants him to taste her food).
So, we're already jumping to the conclusion that he was Jouichirou's nemesis and 3rd seat. But, since 9) and the info that Jouichirou had Souma in a young age (23, since he's from the 69th gen), isn't resonable to think that Azami is older than Jouichirou, by, like, 5 years? In terms of his relationship with Jou, can't we make it something else? We can't close our eyes to other possibilities. Now it's my turn to hypothesize:
What if his relationship with him was more like a Soma vs. Shinomiya? Of course, Shino's a nice dude. But, considering this age gape (5 years), couldn't Jou and Gin's stint at the Elite 10, given their mentality and ideals, and how it clashed with Azami's, a probable instructor/teacher at the academy, made them rivals? We know that the Elite 10 have a huge voice in the Academy's business, something that could have rotten Azami and Senzaemon's relationship from an early time. And since he's a Nakiri and, at the time, the heir to the Academy, isn't it probable that he became a teacher right after graduation? We sure want some kind of rivalry between Jou and Azami, but there's a nice chance they don't have the same age. As I've said, we don't know anything about his age, that's a huge problem to make any prediction. He does have the same level of expression marks as Gin. The "Azami was at the same council as Jou and Gin" solves the question as well, for sure, but it's nice to have some other routes to follow.
So, the post ends here. What I really wanted to say was at the beggining, we can't make too much of what has been shown to us. We should construct the scenarios in small steps (I tried to, maybe I'v failed to do so and jumped some steps too).