Well....obveously if he lives so close HE is the one making the complains which means HE knows which cars are making the noise. My guess is you've been driving like a twat past his house and hes sick of it (rightly so). So either start driving like normal people do, or get your exhaust changed so it doesnt annoy him because otherwise you could be in for much worse things. Consider the fact that he probably COULD have defected you for other things and I think you'll find hes being pretty reasonable. Hell if someone was driving past my house like an idiot and I was a cop, I wouldnt heistate to figure out which one it is and pull them over the first chance I get.
Just cop it sweet or you'll end up in more strife later.