Sometimes foreign speakers can hear the difference native Japanese speakers don't really care about. I have seen speakers of other languages are confused because, to them, Japanese people are pronouncing が and ぜ in different ways. However, this is something ordinary Japanese speakers are not even aware of.
Interestingly, people often unconsciously pronounce one sound in two or more ways, believing it's the exact same sound (allophone). See the "spin vs pin" example in the first link above.
I am a native Japanese speaker, but my ears probably cannot tell the difference between "sampun" and "sanpun" pronounced by you. It's just the same sound, ん, to my ears. So my advice here is "Don't worry". If you care, it may be safe to follow the basic rule, but I don't know how strictly native Japanese speakers are following this rule (again, I cannot hear the difference). You may want to have your pronunciation checked by a native speaker.
To take another example, Korean people say they are generally bad at distinguishing き and ぎ in Japanese. This was confusing to me because, to my ears, they were distinguishing き and ぎ under some rule when they speak their native language, Korean.