Well, I got it figured out. It was indeed being started by X, in this case xfce4. Buried in their xinitrc file (in /usr/local/etc/xdg/xfce4/) ssh-agent was being started, I guess they thought that would be best for their users. Well, I removed that since I like to hand type my passwords over and over. I tracked this down by looking at startx, and noticed the xinitrc file being read in there.
Thanks for the help, hopefully this can point anyone else who is having problems in the right direction.