Well I found the solution myself and post it here in the case that somebody else needs an answer to the same problem.
It appears that the Addresses function in Gnat.Socket returns addresses in different order on windows and on Linux and my code grabbed the address that represents the localhost on Linux but the correct IPV4 address on Windows and therefore it was working on Windows, but on Linux it was now listening only on the loopback device.
So, never rely on the order in which addresses are returned by the socket library