Is man in the middle attack possible in a typical challenge-response based authentication? For a typical reference one can go through this paper.
My question is that is it possible for an intruder to act as server(possibly using DNS spoofing; for the client) and as client for the server and become the man in the middle?
It is not possible to answer your question without more information. "Challenge-response" is a general technique. There are many protocols that happen to incorporate some sort of challenge-response mechanism for one purpose or another. Some of those protocols are vulnerable to man-in-the-middle attacks; some are not.
Generally speaking, if challenge-response authentication is the only mechanism used for authentication, then the protocol is likely to be vulnerable to man-in-the-middle attacks. For instance, one attack against simple challenge-response systems is to wait for the two legitimate endpoints to finish the challenge-response part of the protocol, and only after that is done, start tampering with the traffic.
Generally speaking, to ensure security against a man-in-the-middle, you need some protocol that will provide a secure channel. For most purposes, I'd recommend simply using SSL/TLS. It is well-studied and can provide strong resistance to man-in-the-middle attacks, if used properly.
Update (11/15): It now appears that you were asking specifically about the paper you linked to, by Yang et al. Here is my analysis:
I would not recommend using the scheme proposed in that paper. I have serious doubts about its security.
The paper does not provide a convincing security analysis. The accepted standards in the field today call for a proof of security of any proposed new cryptographic protocol, or at least a very thorough analysis of security against known attacks. The paper offers neither. Put another way, the burden of proof is on the paper to provide convincing evidence that its proposal is secure. The paper does not, I think, meet that burden.
@Thomas Pornin points out that the protocol in that paper will be weak for most parameter choices, or completely insecure for some parameter choices. Thomas explains it well: "If the hash function F has an output larger than p or of the same size, then any eavesdropped exchange provides an oracle for an offline dictionary attack which can rule out half of possible passwords. If the value g generates a subgroup smaller than (p-1)/2, then an eavesdropped exchange provides a much better oracle for offline attacks. In short, they mimic EKE but do it badly. Do not use this protocol."
The paper attempts to "re-invent the wheel". There are solutions in the literature for exactly the problem that the paper poses; they are called password-authenticated key exchange (PAKE) protocols. The paper doesn't seem to be aware of the full extent of prior work on PAKE that had been done before the time of its publication.
There are several PAKE protocols that have been well vetted: if you want to follow the approach outlined in the paper, I propose that you use a well-known PAKE protocol, not the ad-hoc scheme presented in the paper.
Also, the proposed scheme may have an elementary security weakness, depending upon how it is integrated into SIP. The paper doesn't describe what happens after the protocol is completed and each party has authenticated the other. If at this point the connection continues in cleartext, then there is a straightforward man-in-the-middle attack: the attacker simply waits until after the authentication protocol has completed successfully (without tampering with the challenge-response steps), then tampers with messages sent after both sides have successfully authenticated each other. The paper doesn't make clear whether or how this attack is prevented.
