Ditto. I didn't even notice that I'd torn the connector off the logic board until I went to put it all back together. It really is way too tight a connection and since you have so little room to work, it's VERY hard to pull it out without damage.
If you break this connection, anything else you're doing won't matter. Go slow, get a magnifying glass and perhaps try to hold town the part that's connected to the logic board as you attempt to pull out the connector....
If you have any problems or think that you can't do this, I'd say that you should stop, button it all back up and bring it to someone. Even a cheap, used logic board is around $250-350 -- and that's not worth it.