"Because refusing an invitation - even one that is much less sensitive than a sexual proposal - is a more delicate matter than accepting one. The act of inviting someone implies that you hope they will say yes: if they say no, there is a risk that you will be offended, upset, or just disappointed. To show that they are aware of this, and do not want you to feel bad, people generally design refusals to convey reluctance and regret.
Because this pattern is so consistent, and because it contrasts with the pattern for the alternative response, acceptance, refusals are immediately recognisable as such. In fact, the evidence suggests that people can tell a refusal is coming as soon as they register the initial hesitation. And when I say "people", I mean people of both sexes. No one has found any difference between men's and women's use of the system I have just described."
Also, an awesome reaction post, "On the supposed inability of men to understand refusals".