Trying to put a file into virtual memory many times

Thirdeyematt · 10-11-2016, 09:34 PM

One last question however... when I switched the 4tb hard drive to swap Im not exactly sure why, but it now says I have over 400tb of usable swap space when I type in free -t

How did this happen?

pan64 · 10-12-2016, 12:57 AM

If I wanted to give you a correct answer I had to say please post what did you make, how did you add that swap and how did you check that.
Otherwise I need to say it is the same magic what you tried to explain (I can't really make anyone believe in them), but it is just impossible.

JeremyBoden · 10-12-2016, 05:21 AM

Use the

Code:

free -th

command when dealing with large numbers as its easier to read.

Thirdeyematt · 10-12-2016, 06:52 AM

Yes I am just as confused as you... all I did was do:
mkswap /dev/sdf2

Then I did:

swapon /dev/sdf2

It's a 4tb hdd, but after these commands I see more than 400tb swap space when I use the command:

free -t

pan64 · 10-12-2016, 06:57 AM

would be nice to see the result of fdisk -l and free -th

Thirdeyematt · 10-12-2016, 04:46 PM

I guess I was wrong that wasn't the case. I added the 4tb swap and thought I already had 9tb swap, but I actually only had 90gb swap space... So going through the code again:

for i in {1..300000} ; do cp MentalClarity.png /dev/shm/MentalClarity-$i.png ; done

This copy is taking a very long time to fill up my 4tb hard drive, I thought copying files into virtual memory was supposed to be instant? Is there a way to copy the information in the file instantly to virtual memory?

I tired hard-link, which will store the memory in the virtual memory, but not the inode of the memory which saves a lot of time? The code I wrote had an error of could not create cross-device link (I stored MentalClarity.png in ~/MentalClarity)

for i in {1..300000} ; do cp -lr ~/MentalClarity /dev/shm ; done

Shadow_7 · 10-12-2016, 04:46 PM

$ sudo swapon -s

Provides a little more details than free. Since you can have several swap sources with priorities and other properties.

Thirdeyematt · 10-12-2016, 04:55 PM

Quote:

Originally Posted by Shadow_7

$ sudo swapon -s

Provides a little more details than free. Since you can have several swap sources with priorities and other properties.

Yes it revealed the 90gb partition in linux and my 4tb swap drive that I created.

jpollard · 10-12-2016, 05:46 PM

Quote:

Originally Posted by Thirdeyematt

I guess I was wrong that wasn't the case. I added the 4tb swap and thought I already had 9tb swap, but I actually only had 90gb swap space... So going through the code again:

for i in {1..300000} ; do cp MentalClarity.png /dev/shm/MentalClarity-$i.png ; done

This copy is taking a very long time to fill up my 4tb hard drive, I thought copying files into virtual memory was supposed to be instant?

Nope. It always takes time to read the data, and more time to store it, even into memory.

Quote:

Is there a way to copy the information in the file instantly to virtual memory?

Nope. EVERYTHING takes time.

Quote:

I tired hard-link, which will store the memory in the virtual memory, but not the inode of the memory which saves a lot of time? The code I wrote had an error of could not create cross-device link (I stored MentalClarity.png in ~/MentalClarity)

Nope. hard links only work within the same filesytem the file already resides on. Hard links do not copy data. All a hard link is composed of is the inode number and a file name. The inode number is the header of an already existing file. The only thing taking up space is the inode number and name.

Quote:

for i in {1..300000} ; do cp -lr ~/MentalClarity /dev/shm ; done

Copying a file 300,000 times still has to read the data... Assuming a 30MB/second read rate and a small 5MB file, you will be transfering 5*300,000MB or 1,500,000MB. a 30MB/second read would need 50,000 seconds. Now granting that 5MB just might fit entirely in memory buffers, you can go faster...

But don't forget - tmpfs must put stuff in swap as memory fills up - now you are limited to the write speed of the output device. Since you were indicating using USB for that. So now you have to deal with writing the data to whatever speed your flash device can do.

All you are ending up doing is causing a lot of page thrashing...

BTW, the /dev/shm mount is for managing shared memory segments - not data files. Fortunately, system utilities tend not to use shared memory (user utilities, however, do - things like pulse audio uses it.

Thirdeyematt · 10-18-2016, 02:28 PM

Ok well thank you for helping me, that actually worked very well. However, after making the external usb 2.0 hdd into swap, and mounting in /mnt, copying files to the directory makes the computer very slow... Is this because the drives are usb 2.0 and have slow transfer rates?

Would the computer not lag if I partition my 10tb hard drive 8tb/1.1tb I could use that 8tb as swap space and it would have a lot faster data transfer rates. It's just a regular internal hdd 7200rpm

jpollard · 10-18-2016, 06:23 PM

It will go faster... but any time you start thrashing the system it will start going slow.

I have 8 cores, 8GB RAM, and 16 GB swap. But if I accidentally push my POVRay (a ray tracer) into that much memory - the system nearly dies. If I do it deliberately, then I just have to put up with it.

What you are doing is filling physical memory - then the system has to allocate swap space, and copy the pages to disk in order to free up physical memory for more data... which you then fill up, forcing the system to repeat. So read data, force writes, to read more, to force writes....

You would accomplish the same thing just by mounting the space used for swap as a normal filesystem and just copy the file there.

Still a waste of time since from all indications you are using the same file every time.