did useradd -s /sbin/nologin user1. then added /usr/sbin/nologin to
/etc/shells, but it still won't let those users ftp in. normal users
work fine. shouldn't this work? any help is appreciated

debian 3.2.0-4
vsftpd 2.3.5-3

Yes, it should work that way. It will be good if you share your vsftpd.conf file so that we will get better idea of the issue.

listen=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES
local_root=/var/ftp/pub
listen_port=21
pasv_enable=YES
allow_writeable_chroot=YES
xferlog_file=/var/log/vsftpd.log
ftpd_banner=welcome
deny_email_enable=YES
banned_email_file=/etc/vsftpd.banned_emails
chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/chroot_list
secure_chroot_dir=/var/run/vsftpd/empty
pam_service_name=vsftpd
rsa_cert_file=/etc/ssl/private/vsftpd.pem
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=YES
ssl_sslv3=YES
require_ssl_reuse=NO
ssl_ciphers=HIGH

Did you try creating a normal user and then changing the shell to /sbin/nologin or /bin/false by editing /etc/passwd or using usermod. Do you see any difference when you set it to /bin/false instead of /sbin/nologin.

same thing, it won't let you in. this worked in the past. had it setup this way

As you said it worked in past what changed you have made to the system since then. Any changes recently performed which resulted in this. When you try to login with the user who have got shell set to /sbin/nologin what is the exact error message that you get? Did you try taking packet trace. If yes, what do you see in packet trace?

it says fail login

You are not answering to all the queries, without the details how we suppose to know what's going on? You said failed login but what about the error code and what about the other questions that I have posted?

set it up a while back and i'm setting it up again because i have a need again. login failed is the only message i get

Do you have config file from the previous setup. Did you compare the previous and the current config file. As you said you are setting it up again that means it did not work in past. I mean this is a new setup and it never worked with /sbin/nologin. The best way I can think of is to first compare the configs files from previous setup with this one. If that doesn't give you any clue then go with packet trace to see if you find something there.