PS3 Media Server
may fit the bill.
It's a DLNA compliant media server, and I have used it for streaming media from my computer to my Samsung Smart TV (SD, HD 720p, and HD 1080p all three). It can do transcoding of files (although I've never used it, since Samsung Smart TVs support MKV by default), but be warned: PS3 Media Server requires a JRE (Java Runtime Environment), MEncoder, and (possibly) FFMPEG. The program requires no installation as well, as it just runs out of a folder. Looking through the options in the Transcoding Settings tab, it seems you can also specify extensions to skip or force transcoding as well. Hope this helps you out.