Isn't it rude to ask "why" when declining an invitation?

Yes it is rude. A simple "No I am sorry I cannot come to your party etc." should be sufficient without an inquisition.