I think it has more to do with him not having the same set up and exposition before the Culling Games like Higuruma, Kashimo, Angel and Takaba had, and instead he just came kinda out of nowhere, which makes some people seeing him as a less important deal and just a random villain of the week type character that stalls the plot until he is beaten. Plus some people wrote him off as fodder because of his silly outfit ( but they couldnt have been more wrong)
But i don't have any issue with that tho. This arc is meant to be some kind of grand scale death tournament, and it would have been extremely weird and unrealistic if the characters don't get into any encounter with some "randoms" on the way, and instead just rushes to the main objetives.
Plus, this fight has gave us new layers for the purpose of the Culling Game and we are yet to see what the aftermatch of it is going to be.