I want my Inbox to only have people on my contact list.

What distro/version are you using and what email client?

Off the top of my head, I don't know of a way to filter emails into the inbox, only ways to filter them out. How the filtering works depends on the mechanics of your email client.

This article might tell you more: https://en.wikipedia.org/wiki/Email_filtering

I would make a filter with all people on your contact list and divert those to certain folder.

If you use Hotmail or gmail, and probably most others, you can choose to only receive email from those on your contact list. Everything else goes to the spam folder. This is set up in your profile, accessible through the web client.

If your mail client has filters, look for an option like 'not from:', put a mailing list with all your contacts in there, and have the filter put results in a spam folder.

I've stopped using mail clients for personal email, I just use the web client, or a mobile app. At work we use MS Outlook, but our enterprise does spam filtering well.

If you're setting up a mail server on Linux, you need to specify a bunch more details.

If the OP is managing his own mail server, spamassassin comes to mind. My spamassassin filters are so aggressive that some legit mail ends up in the trash or spam folders.

If the OP is using an email client (MUA), then yes, he would need to tell us which client to get help. [or look at the help for that client]
For example, I have extensive filtering in Thunderbird to sort incoming mail into folders. I see that TB has the capability to set up a filter that says "if From: isn't in my address book: Personal address book then Move message to <some folder>"