If I'm reading the vsftpd manpage
correctly, to limit access to specific users, you set userlist_deny to NO and then list the users you want to allow in the userlist_file file.
To get them all in the same directory, it looks like you need to set the local_root to a directory. Again, if I'm understanding the manpage correctly, local_root forces all non-anonymous users to that directory.
As long as you leave the write_enabled set to NO, users should only be able to download, not upload.
If you want the users to not have a home directory, you can create the users with adduser -M, which prevents a home directory from being created.