Every character in HXH has an implied physical body rating. Aura is still only a multiplier on your physical body. The standard aura punch is at best twice the physical strength. Using a special might get you 5 or 10 times the normal power, but it's still a multiplier on your physical strength. If you've say Kuroro use Big Bang Punch, it's not going to be anywhere as strong as Ubogin, not only because he is the wrong school, but that clearly he is physically weaker than Ubogin as well.
Now why does the manga say you can't beat aura without using aura yourself? Because it's implied the physical difference between comparable characters are very small. Let's say Ubogin's physical power has a rating of 10. We know Kurapika can hurt him when Ubogin cannot use Aura, and Kurapika has at most a 160% modifier on direct punch. This means Kurapika's physical body is no weaker than 10/(1.6) = 6.25. They're probably a lot closer than that as Ubogin always thought of Kurapika as a worthy opponent. A corollary of this is that it takes a very small difference in your total power to do serious damage to the other guy. Therefore, not using aura is pretty much suicidal against anyone who is remotely as strong as you are. Note that this is not true when there's a wide disparity in power. When Gon faced Pitou the second time, Killua said Pitou can no longer defeat Gon without using aura, which means he can do it the first time. After all, the Ants are allowed to have off the scale physical abilities, so if Pitou is say a 50 and Gon is a 10, then he is very safe even if he never used aura.
While muscle mass has no direct correlation to the hidden physical power, people who are very physically powerful typically are drawn to be muscular. Ubogin and Razor are obvious examples. Yupi is drawn as the most muscular amongst the royal guards and it's nearly certain he's also the physically strongest member of the three. Of course, Meryem is rather skinny and his physical power is off the charts, but he's clearly an exception to this rule.