Isn't it rude to ask "why" when declining an invitation?

You never have to explain or give reasons ever for declining any invitation. None of them are mandatory and the less you say the better, unable to attend is plenty and if they get offended tough sh*t.