When streaming Ogg/vorbis to xmms via http the url needs to end in ".ogg". This could be done by providing a filename with the sout line, something like this: vlc --sout http/ogg:<network interface>:port/filename.ogg
As a temporary hack you could append "?.ogg" to the url you give xmms.
XMMS supports es, ps and mpeg1 muxers with mpga audio out of the box. The ts muxer gives strange audio in xmms.