I'd add that if I did have it and it was his I would tell him he's welcome to come get it.
That's a much better solution than letting him manipulate you and damage your happiness. If he really wants it back, hand it over to an FFL in your state, and let him carry the burden of transferring it. As you undoubtedly already know, you cannot simply hand it to him.
