BSOD Before Installation

AMD Phenom
4GB RAM (2x 2GB)
BioStar TA770 A2+
Nvidia GeForce 9800GTX

SATA 0 Western Digital HDD
No IDE drives

First of all...

1. Unbuntu installed without ANY problems and ran fine
2. Tried to install XP as well but it would freeze at "Starting Up
Windows" during installation.
3. Removed 1 DIMM so I'm left with 1 x 2GB Dimm

I didn't really configure anything special in the BIOS until now.
Insert the Vista x64 DVD into the drive, asks for CD boot, copies files
to hard drive and after it's finished I get a BlueScreen with
Machine_Exception error.

Do I have to configure anything special since it's a Quad Core or
because I don't have any IDE drives and only SATA ones?

I build computers before and always had a problem or two but always
solved them. Hoping to solve this one as well. Thanks :)