Ok, I got it fixed and hopefully this problem won't return, Below are the steps I did:
I set the date to the correct time:
date 03211323 <-- this set my date and time to mar 21 1:23pm
Then I checked hwclock by doing:
hwclock --show
Then I had set the hwclock to what the system has:
hwclock --systohc <-- this command sets your hwclock to what your OS is set to.
I've been checking the time for a few hours now and everything looks good, We'll see how it goes when the day changes at 00:00.01 tomorrow. Thanks Sixpax!