Linux desktops are biased toward maintaining cache in memory - at the expense of potentially swapping some application(s) anonymous memory.
There are a few (crude) controls to alter this. Be aware that what you perceive is not necessarily the whole story - you see the end result, not the decision making processes (plural).
The most obvious knob is "swappiness" - this is a recommendation to the system about how you want things managed. 100 says always prefer to keep cache in memory - 0 (zero) says always try to minimize swapping. The default is 60 - can be seen by
You can adjust its behaviour on the fly by echo'ing a new value to that control - maybe try 30 ... or 10 ... or 0. Depending on your distro can be hardened via sysctl.
In addition to this, dirty file writes are managed by the I/O schedulers - they use different algorithms to consolidate/optimize I/O. This includes delaying writes - for some seconds in some cases. There are individual controls per scheduler, and the scheduler itself can be changed.
Then there is the specific filesystem itself, and its block-level driver.
And any hardware (RAID) or software (LVM, mdadm ...) that happens to interpose itself.
All potentially affect the rate at which I/O is (physically) written, and consequently how fast storage (file cache in this case) can be released to the free pool.
There is also some "hidden" issues with memory allocation - the slab allocator has had some serious work in recent kernels. Even just upgrading the kernel might help from your kernel level.
As you can see, this is not a trivial "where can I go to look at the code" sort of thing. You will be all over the place.
/proc is a window into kernel structures - it is always current at the instant it was read, as the data only exists if (when) it is read. vmstat reads /proc "files", so the same applies, although it does some averaging.