You are leaving a lot of questions unanswered.

You have to be a lot more specific as to what will not work.

You need to have 5 RCA cables going from the 5 OPPO RCA analog outputs to the Marantz receiver's five analog inputs, and the 5 preouts of the Marantz connected to the 5 power amplifier channels through 5 more RCA cables. That is 10 RCA cables, which you don't seem to have.

The SW output jack of the OPPO can be connected directly to the subwoofer input jack (that makes 11 RCA cables), or the sub can be connected to the receiver.

I have no idea what an XBA connector is. Do you mean XLR? The XLR balanced outputs of the OPPO are stereo only and will only give 2-channel sound. When you want 5.1, they are not in use. They are only used to send 2-channel audio to the receiver when playing audio CD discs (assuming it has XLR inputs).

If you have a SACD disc which has 5.1 sound, the 5 RCA analog outputs should definitely give you 5.1 sound from your receiver, assuming it has been set properly using its menus, and the 10 cables mentioned above are connected.

Playing a DVD with 5.1 sound or a program with 5.1 sound from Netflix etc. through the WIFI receiver of the OPPO should also give you 5.1 sound, if available from the program. This must come from the 5.1 RCA outputs of the OPPO to the receiver (the OPPO uses its own DAC internally to generate those 5 analog outputs to the receiver).

You should have an HDMI cable from the OPPO's HDMI 1 output to one of the TV's HDMI inputs (HDMI #1)to send the video to the TV when viewing a DVD or Bluray disc (the TV audio should be muted).

When viewing content from a cable box or satellite box, you want to send it directly to the receiver with a second HDMI cable, and then a third HDMI cable needs to go from the HDMI out of the receiver to one of the other HDMI inputs of the TV (HDMI #2).

Use the TV to select the input in use.

The only reason to use one of the digital inputs of the OPPO is if you want to send content directly from the internet or computer or some other device with a digital output. This may or may not be 5.1, depending on the source and the content.