First thing I would do is check to make sure there isn't any debris in the jack. Things that get carried in pockets tend to get lint in them and such. Try hitting it with canned air (being careful not to blow moisture into it) and also *carefully* swabbing it out with something long and pointy.
After that, the next thing I would check is the metal leads that make contact with the headphones. If one of those got bent somehow it might not be making a full connection. If you're luck you might be able to pry it out with something so it touches better. This is just a guess though, I don't have a 4th gen nano so I don't know for sure what the jack is like.