If you're knowledgeable enough to compile things, I did the work necessary to get Audacious 2.4.0 building and running on OS X back when that version was still current. That will play a large handful of formats including whatever the ModPlug plugin supports, and a handful of consoles. (In addition to the usual MP3/AAC/Vorbis)
I posted instructions along with my source modifications to their forums a year and a half or so back, but they switched forums recently so the post is no longer available. The source code to 2.4.0 with my changes needed to build it on OS X is here:
Short Summary which won't help if you don't know what you're doing:
1. You need to install the dependencies using port. Use +universal variant for all packages that have them. I use the +universal +no_x11 +quartz variant for installing GTK so that Audacious won't require X11 to run. I don't recall all of the dependencies though, I had most of them listed in the original port. For most of them you'll be able to figure them out because configure will usually error on stuff that is missing.
2. Grab libmcs 0.7.2 and libmowgli 0.7.1 from the official site. Those don't require any modifications to build from the site. Audacious needs them to build.
3. Use the following command when running configure, to force it to build 32-bit (64-bit doesn't work) and tell it where to look for the stuff installed by port:
followed by make and make install to install the plugins.
6. The audacious binary that was built when you compiled audacious can be manually placed into an app bundle to launch like a normal app. A pre-made bundle minus the executable can be found in the contrib directory.
I've used Audacious since 2008, so these instructions should work all the way back to Leopard. I last re-compiled 2.4.0 when Lion came out, so I know these instructions work for Lion.
The latest version of Audacious is 3.1.2, which I haven't bothered to upgrade to because 2.4.0 works fine for me and I didn't want to go through all of the work required to get it working again. It seems at one time there was a developer who actually worked on making it compatible with OS X, which is why there is a CoreAudio plug-in, but OS X support seems to have been abandoned years ago. The CoreAudio plug-in wouldn't even build on Snow Leopard and Lion without some modifications, and barely works with the hacks I threw in to make it compile. (The plug-in wasn't even compatible with 2.4.0 yet they included it with the source) You need to set the "Bit Depth" option to 16 bit for the CoreAudio output plugin, the rest of the options will crash. Also the configuration window for the CoreAudio plugin will crash as well. But the rest of the plug-ins and configuration for those work fine.
I wish I could package up a pre-made build to distribute, but there's no easy way to integrate all of the dependencies and plug-ins into a single app package. Once you get it built, it works fine though. And it supports WinAmp 2.x skins, which is a plus.
m4a support works fine, you just need to make sure the plug-in for it was built when you compiled plug-ins. If the configure script doesn't find the dependency that the MP4/AAC plug-in needs, it won't be included.
I believe libaac is the dependency you need to install for that plug-in. If you run configure again for the plug-ins, you should see what it was looking for and what it didn't find for the reason why you didn't get that plug-in.
The Ogg Vorbis / FLAC plug-in has the same issue, dependencies are needed. For that one, I believe you actually need to download mp123 manually, compile that, and even after installing it you need to manually feed it to pkg-config so the plug-in configure script will find it.
I know its been weeks since my last post but I've got good news! I finally got it to work with mp4, ogg and mp3! Audacious with aac support needs faad2-2.7(latest version) and for mp3 support mpg123-1.13.4 (latest version). The only issues I'm having is that I find it extremely buggy, like it froze when I tried to stop playing an mp3, and I can't get into the perferences menu with the winamp skin enabled. It let me pause, but then I double clicked on the interface and got "Bus error: 10". I wish it was less buggy and if there is anyway to fix them then that would be great, if not than oh well, this still has been fun! Anyways I'd like to thank you again.
I use only the Winamp version of the interface, and the UI portion of it works fine.
Sometimes if the window doesn't have focus, I need to click something like the play/pause button twice for it to work, but I have no problems right-clicking to pull up the menu and load files, etc.
One thing I did notice over the years was that the behavior of it varied a lot depending on the GTK version. Back in 2009-2010 I had a lot of issues with the file open dialogs for example. Early on, the row selection wouldn't redraw when I clicked on a file/folder in the dialog, then a few months later when I updated to a newer GTK version the file dialogs weren't sorted properly, it was like they were comparing alphabetically by length so short file names came first, etc.
The build I made when I upgraded to Lion has been the most stable of all times I've compiled it. Aside from the sticky buttons, I don't think I've even seen it crash more than a couple times in the last 6 months, and I use it almost every day.
In other words, it seems like it's just luck. At some point I'll see if I can roll a better solution for a more recent version, but right now it meets my needs so it may be awhile.
I think I've found an easier solution which is to grab the latest Windows version of Audacious, install wine-devel from mac ports, then run wine audacious.exe, which requires X11(Comes preinstalled with Lion) there seems to be no crashing bugs, and this should work with just a vanilla wine install, but I haven't tested it. pic: http://dl.dropbox.com/u/38348368/winveroflinuxmediaplayerinosx.png