Despite being zero, a Fairy's physical capabilities obviously fluctuate depending on the story. King got knocked out by Cain, but later on he was capable of taking a crap ton of physical damage in Liones from the Sacred Tree's Roots and get back up like ain't shit happened. Same when he was going against the Albion.
That overall 50k literally doesn't mean shit if he's fighting someone hand to hand, without magic and Basquias, he'd lose every single fight
Why would you take away his main source of strength though? Match ups matter, but not when you take away the aspect that makes them strong in the first place. That's be like me saying Galan would lose in a fight against Vivian, if he didn't have his physical strength and his weapon. He'd lose every time. Of course he'd lose every time because I took away his main way to fight.
Match ups matter when you put someone up against someone or something that can nullify their ability at full power or just overpowers them completely. Gloxinia will destroy Gilthunder, despite Gilthunder having a higher physical ability.
A better example of a bad match up would be Galan and Ban. Despite the fact that Ban wasn't able to finish the job, he abilities allowed him to weaken Galan's primary source of strength and add it to his own. Coupled with his immortality, Ban is Galan's worst match up.
Someone who would actually be a bad match up for Gloxinia would probably be Deathpiece, whose Melody magic would screw with Gloxinia's timing on using his magic. Or Arden, whose abilities make it hard for someone to actually use their magic effectively.