Uh, I just realized if you uncheck the box, it does unfavorite. However, THAT is very confusing. The Unfavorite button should simply remove the favorite action without interaction.