It doesn't sound as though you have the funds to waste on a B visa application that is likely to be denied. Nigeria does have a high rate of refusals (46.51% last year), but that does mean half of the applications received are approved, so there is some chance - but only if he has really good ties to Nigeria. And realistically, with you in the picture, that reduces his chances further, so it's your call as to whether you try or not.
But how about a third country - Canada maybe? Again, he may not get a visa, but it would be worth a try and you could potentially drive or get the train there instead of flying?
Either way, you're going to have to meet, and likely several times for him to be granted a visa. So I'd start working out a way to do that and then take it from there.
Best of luck.