For me, \small works; I'm putting an example where I used it below. The effect of \small is not very strong; use \tiny instead to see a big difference. You see my other efforts to squeeze the table: reducing table column separation to 5.8pt, and suppressing the (half) space before the first and after the last column with @{}.
Code:
\begin{table}[htbp]
\begin{center}
\tabcolsep 5.8pt
\small
\begin{tabular}{@{}l|llllllll@{}}
& bott & c & c2 & dk & edif & font & polt & trat \\ \hline
bott & 0 & 0.3957 & 0.3910 & 0.1553 & 0.2064 & 0.2588 & 0.1759 & 0.2079 \\
c & 0.3957 & 0 & 0.0159 & 0.5251 & 0.1811 & 0.5255 & 0.5381 & 0.2388 \\
c2 & 0.3910 & 0.0159 & 0 & 0.5296 & 0.1824 & 0.5386 & 0.5094 & 0.2412 \\
dk & 0.1553 & 0.5251 & 0.5291 & 0 & 0.3133 & 0.2071 & 0.2779 & 0.2884 \\
edif & 0.2064 & 0.1811 & 0.1824 & 0.3133 & 0 & 0.3237 & 0.2830 & 0.1132 \\
font & 0.2588 & 0.5255 & 0.5386 & 0.2071 & 0.3237 & 0 & 0.3364 & 0.3267 \\
polt & 0.1759 & 0.5381 & 0.5094 & 0.2779 & 0.2830 & 0.3364 & 0 & 0.3239 \\
trat & 0.2079 & 0.2388 & 0.2412 & 0.2884 & 0.1132 & 0.3267 & 0.3239 & 0
\end{tabular}
\end{center}
\caption{Dissimilarity between the descriptors and hence between the shapes
of any two of the small test objects.}
\label{tab:dist8}
\end{table}