Start by cleaning out the computer inside and out. Blow out all the dust and make sure all the fans are running. Then a scan disk and a defrag for the inside. Then i would look at the Ram remove the added ram and just run one Dimm. run the machine and see if it gets more stable. If these don't work then. We start on the video card try running this
www.ozone3d.net/benchmarks/fur/
It will stress test the Video Card if it fails then you know where your problem is.
After that i would look at power supply.