As far as I’m aware there very few real-time 5.1 (Dolby Digital / AC-3) encoders for the Mac that integrate into the default CoreAudio HAL (and can thus be used by any application); I’ve only stumbled upon ac3jack, which builds on the multi-platform JACK Audio Connection Kit.
I thought it’d be interesting to learn something about CoreAudio and low-latency coding and thus decided to write such a system. Currently, this is split into 3 components:

• CoreAudio AudioCodec for transforming LPCM into encoded AC-3. I’m using libavcodec as the encoder for this.
• Audio driver presenting a virtual sound output to the system, and forwards that data as a provided sound input. The idea is to select the virtual output as default / system output, and then encode from the virtual input into the compressed format and forward that to a physical optical output.
  In 10.8 “Mountain Lion” you can write these sorts of plug-ins as sandboxed user-space components, which is quite neat (but not well documented at the moment).
• A normal application / menu extra that takes raw sound from the virtual sound input, passes that to the AC-3 audio encoder, and then forwards the resulting stream to a physical digital output port.

The encoder seems to work; I’m currently working on the driver and then comes the user-space connector / configuration application.

Fun times!