I want clarity on the whole death to infidel thing?

so is it just people who danger the teaching of Prophet Mohammed, and the Qur'an, or people that refuse to convert to islam??