Are all the sockets active? It's surprising that your server would completely bog down as you approach the limit unless it's an OS issue.

Are you polling them in a tight loop? If so, that's probably a bad idea. If you're not already, try using something like select(), which blocks until some number of your sockets are ready for reading. When it returns, you service those sockets that are ready, then call select() again.

Check out the man page. (I'm assuming OSX comes with man pages.)