Losing her mother is part of it too. The biggest thing about robins story is the cluster of things she was forced to face and the sheer amount of things she lost which translates to everything. She was maltreated by her guardians, hoped for her mum, got to meet her and lost her on the same day as well. Add to every other thing she faced, it becomes an impossible story for a little kid. So its not about picking one particular aspect of her tragic past, its having to endure all of that at once.
Also the oharans might not have treated her as one, but that was still part of her home. And she watched an entire ship of the residents get blown up right before her eyes and considering she was about to get on too. Its enough to shake a little kid hated or not.
Robin's heart was definitely wounded if not scarred ... and we have seen her trauma of buster call during Enies Lobby ... But I think that was the trauma rather than the loss... She lost everything on that island which she never had anyways but just had hopes... She was considered a devil in her town ... Children won't play with her ... she was out of choices to be blunt ... Compare that to Nami , where she had to forcefully throw everything, her childhood for the sake of her village, when she had a choice to abandon them and live her own life.. ... compare that to Brook who lived in agony all along, and the time he spent there on that ship was long enough for someone to abandon his hopes which he didn't... and considering after their disaster how they have lived there life after that ... I would not rate robin so high on my charts with the girl she became later ... So at the end of the day , what robin lost was the hope ... and hope she found in Straw hats ...
Saul also said the same thing , that you will find a place where you will be loved ... All this time Aokiji kept the tab on Robin, like an angel in the shadows... He appeared in front of Shs even , because of Robin only... and at the end of Enies lobby he was shown relaxed, considering robin found a place she is loved at ...
There are other Shs who carried the guilt all these years, like Sanji, Franky and chopper .. and that guilt can never disappear even they have a crew and a purpose in life, its always there ... ... which could be considered a sad back story ... Robin's wounded heart got healed after coming up from the fear of buster call finding the nakamas ... Think about Sanji, Franky or chopper, they are scarred for life with the guilt, it was because of them for what happened to Zeff, Tom or Hilluluk(Hilluluk case can be said weak as he was anyways about to die)... but whatever