Finally managed to get CoreAVC + Subtitles + Avisynth/Mencoder working for the PS3 Media Server on Windows 7 RC.
By right, the default usage of just Mencoder for transcoding with subtitles is just fine. However, I have some anime files that don’t size/zoom just right on the TV (i.e. there is no Zoom screen mode on the PS3, instead Normal mode displays like Zoom mode). AviSynth/Mencoder displays them fine though, with all 3 screen modes available, but of course, no subtitles. And since I have CoreAVC with CUDA acceleration, might as well make use of it for the other 1080p files.
So yes, follow the instructions for CoreAVC in PS3 Media Server’s README/FAQ files.
Then implement this hack to stop Windows 7 from using its built in DTV-DVD filter:
Replacing the Default DTV/DVD Decoder without Breaking Live TV Playback
(Take note of the different registry path for 64-bit in the comments)
Now, you should be set. I’ve tested this so far with MKV files that have built in subtitles. With vsfilter.dll in the Avisynth plugins folder, I’m expecting double subs to appear for movies with external subs (this is my assumption by logic, but might not be the case), so removing vsfilter.dll from the plugins dir might help with that.