Why specifically mention cat? There may be, and are, better ways. Try:
num=( $(<test.txt) )
...Which uses only bash's built-in file redirection. The input file will be broken up according to your IFS settings (space/tab/newline by default). Each "word" in the input file will be stored as one entry in the array.
It would be nice though if you could define what you have a bit more clearly. Are all the numbers in the file on one line? Is there anything in the file except the numbers? Is it an extremely large file?
PS: bash v.4 also has a new mapfile/readarray built-in, which will populate an array with the contents of a file; but it only works on a per-line basis, afaict.