Now you're starting to provide info that would let someone help you. But it is still unclear what you mean by "1 gb of memory used" and it is unclear that 1GB is unreasonable for what you had open at the time.
The amount of memory used inside X reasonably depends on the GUI activities of all the programs you are running. But it may not be reasonable to expect X to release that memory back to Linux after it is no longer needed. The boundary between the OS and a process for memory allocation is complicated.
If there is no memory leak in X, then memory X keeps after a large GUI demand has gone away should be reused by X (rather than allocating still more) the next time a similar level of GUI use is present.
If there is a memory leak in X, then (assuming you have adequate disk space) having more swap space than you would otherwise need could postpone any significant consequences of that leak for a long time. If the real memory is needed for anything else, Linux is very good at moving leaked memory out to swap.
Even without an actual leak in X. If it is simply hanging on to most of what it needed from the last high water mark of GUI use, if you switch to some less GUI intensive but more memory intensive computation, you may need extra swap space to avoid problems.
In a popular version of a popular distribution, I would expect any real memory leak in X to already be well investigated. But X is one of the most complicated parts of the distribution for memory leak issues, so maybe you have some combination that involves a memory leak that hasn't been found and patched.
I'm primarily a Windows user, so I just live with the fact that GUI resource leaks gradually drag down a system and eventually you need to reboot to clean them out. In my own experience, Linux is better than that. But I see plenty of info online showing that other people have problems with GUI resource leaks in Linux. I'm sure X developers and Ubuntu developers take the problem seriously. For an individual user, you might want to ask whether the problem (that you might or might not actually have) is worth the effort that would be needed to figure it out. But if you think it is worth the effort and you either force it to happen or wait for it to happen and then gather good info about what is happening, I (and I'm sure more expert others) would be happy to help you interpret that info.
Originally Posted by Foxbat1155
If I tried to stop that process to see what happened it would restart Ubuntu, not an actual pc restart.
You are talking about stopping the whole GUI system of Ubuntu.
I have stopped and restarted the whole GUI system in older versions of Mepis, Debian, Ubuntu and other distributions without rebooting many times in order to tweak display settings for support of obscure obsolete display hardware. But I haven't done so recently and the details tend to change a lot across versions.
I'm sure you could learn to correctly stop and restart X without rebooting Ubuntu and I'm pretty sure that would clear the excess memory use as if you did a real reboot. But none of your GUI activities could stay open across that, so I can't see how it could be any easier or better than a full reboot.