Use tweezers to grab the white sides of the trackball bracket and remove from the phone.
If this method is unsuccessful then place the device over your hand (with the screen facing downward). Make sure that the trackball is positioned directly over your hand. Now turn the device over carefully and the trackball will fall out. That will be the only thing that is loose so make sure that you are able to catch the trackball when it falls.