Call him up and matter-of-factly explain that you don't think the job is right for you, so thanks but you'll have to decline the offer. Then, ask him out. With all that conversation already it will be like a second date.
He was very talkative! As am I.
No you hang up, no YOU