Lets start with your 2nd and 3rd issues as they are tied together.

The "side" speaker outputs are really your surround outputs in 5.1 and the "rear" outputs will only output sound in 7.1. Switch the rears to the sides, and this will resolve both your sound output issue in any 5.1 DSP and it will resolve your Audyssey issue. The rears are only active when the side/surrounds are active so if those are not making sound, Audyssey will error out.

Incidentally, those rear speakers should be set a wider distance apart than the front speakers. That is the optimal way to set up a 5.1 system. Audyssey will treat them like surrounds and will deal nicely with the fact that they are rear facing instead of side facing.

Regarding the turntable, you should not be plugging your external phono preamp into the phono input. If you are, that will create problems because you would be plugging a phono preamp into a phono preamp. What surprises me is that you are having issues when you inserted into the VCR or CD inputs. Double check and make sure you have the isolate switch set correctly or the 20/100 switch set correctly if you have an MC cartridge.

Final test is try the internal phono preamp on the Marantz and see if your turntable is working with it assuming you have a MM cartridge. If it works well with the internal phono preamp and ground on the amp, then the problem is going to be with a setting on the phono preamp.

I know this sounds odd but I find my turntable sounds flat in Stereo unless I am running it through direct or "pure direct" mode. Try Direct or Pure Direct through these other inputs. In stereo, I think the EQ is on and if Audyssey has errors out, it could make it sound exceptionally weird. The one thing you will need to do is turn the subwoofer on for both modes as I think the default is for it to be off.

The reason this matters is because if Audyssey is in, Audyssey is a digital EQ. That means the Pre / Pro is digitizing the analog signal, applying the EQ and converting it back to analog. This will not sound good. Direct and Pure Direct is a quick way to test if the EQ a is screwing up the sound.