Who would you be "letting down" if you didn't go though? It's not as if it's just one friend you would be "leaving in the lurch" as a group of friends are going.

Would a true friend stop engaging with you if you didn't go on a planned holiday with them due to circumstances? Is telling them the worst bit?