There are a tons of tips and tweaks out there for optimizing iTunes and your Mac's various settings for audio playback that can have a big impact. Google around. These will help. And they're free. Start there.     

That said...

I suspect it's the HDMI. Generally not a good option for dedicated 2-channel audio output because HDMI was designed to carry a bunch of other signals and data.  I would take HDMI out of the audio equation.

There are a couple of different routes and they will all cost you some bucks:

1.) Use the Mini-Mac's digital output. It's the same output where the headphone jack is located. You'll need to buy a Mini TOSINK cable and run that into your Denon's optical input. You'd also need to adjust some settings on your Mac, but I'll assume you can figure that out. Cheapest option.

2.) USB Converter. Something like Musical Fidelity's V-Link will take audio from your USB and convert it to either Optical or Coaxial TOSLINK that you can hook up to your receiver. You can buy one for a hundred bucks on Amazon. And of course the USB cable and the TOSLINK cable - but it sounds like you may have a few of those cables hanging around. I had a V-Link, it sounds great and works like a charm. 

3.) Buy a seperate USB DAC. The DAC in your Denon, again, wasn't designed with 2-channel audio in mind. There so many DAC options right now I won't get into it. But this will probably give you the best sonic results. If you don't want to spend over $100 - get the V-Link.   

Cheers.