Here's my experience with streaming to PS3:
MPEG2 files have no size limit. Audio should be AC3 or LPCM.
M2TS files have no size limit. You can stream anything that's MPG2 or AVC. VC1 doesn't work. Audio: AC3 and LPCM works. TrueHD,DTS doesn't work.
So i suggest you convert 720p and 1080p MKV files to MPEG. I use Pavtube. Where appropriate I'd like the video file to simply be copied (as the freeware program tsMuxer does - saving a lot of time) and have DTS audio converted to AC3 so it works OK on the PS3 (which tsMuxer doesn't do).
http://www.pavtube.com/guide/convert-mk ... o-ps3.html