Yes, I restart the service after every config change. Notifications are being delivered to root's mailbox.
One other thing I forgot to mention. I tried using another smtp server as a relay, but I was receiving connection refused messages in the nagios log despite the fact that I was able to connect on port 25 and send mail without issue.
# Just one contact defined by default - the Nagios admin (that's you)
# This contact definition inherits a lot of default values from the 'generic-contact'
# template which is defined elsewhere.
alias Nagios Admin
service_notification_commands notify-by-email host_notification_commands host-notify-by-email
# We only have one contact in this simple configuration file, so there is
# no need to create more than one contact group.
alias Nagios Administrators