You are looking for process substitution
comm fileY <(cut -f5 fileX) > fileZ
This is bash specific, so make sure you have #!/bin/bash
at the top of your script and not #!/bin/sh
. Most programs use "-" as a name for stdin, so you can also do
cut -f5 fileX | comm fileY - > fileZ
That is not bash specific.