It could be handled by telling him that in light of your wives being friends you would like to remain socially cordial but be up front and tell him you have considered calling Colorado F&G on him. Just tell him you are not a poacher and that while you can be neighbors on good terms you are not going to jail with him and not putting yourself at risk and suspicion by hunting with him. And tell your wife what you told him so she doesn't get blindsided.
I think this is the best suggestion.
