I think he is a virgin! I just think when the time comes to actually sleep with them he just ends up leaving because he secretly has respect for women(due to taking care of his mother for so many years and being in a village that mostly made up of women due to men being sent to Baal's dungeon...