He wants us to invite another couple over for game night. I laughed. He did not. Now what?

We have been together five years and the sex is good, so this is not about that. Last month he started saying we should try something different, and last week he named the couple. Our neighbours. I laughed, he went quiet, and we have been polite since. I am not interested and I am afraid of how to say it without sounding like I judge him. How do you say no kindly?