I hate Old Man Frey as well, but then again, it is all a part of Martin showing (rather than telling) what a rotten king poor Robb had turned out to be.
Two of the huge underlying themes of the series are honor and leadership. The honor is explored through the fates of Ned, Robb, and Jon, and the leadership is explored through the action of all of the ruling Kings and Queens.
Robb not only let his troops run amok in the raping and pillaging of the Riverlands ("Wolves or Lions, makes no difference") - lands he was supposed to be leading and protecting - but he put his personal honor above his sworn word to his highest vassal lords and allies. In doing so, he proved himself to be completely untrustworthy as far as they were concerned.
They acted on that.
So while it is easy as a reader to get caught up in "The King of the North!!!" and the romantic legend of a young King and warleader who was defeated while never losing a battle, in the overall series plot he was an example of poor leadership, poor kingship, and his strict sense of personal honor and valuing that over his dealings with his peers ended up killing him.
