Extended memory with swap in ARCH (swap to 30GB)

lilsirecho · 2006-10-13 21:27:28

Reduced the RAM to 1GB.

Repeated the transfer tests as before with hdb1 as swap priority=3 and set to 25GB in /mytmpfs.

Results:

Up to 16GB of transfers, the transfer rate :

Up to 10.9 GB rate was 28MB/S to 31MB/S....

From 10.9GB to 16GB rate was 22MB/S to 30MB/S...

The system ram free at end of transfer is 19MB.

I address this post in that condition.

Next, will try 2GB RAM.

lilsirecho · 2006-10-13 22:09:24

With 2GB of RAM:

Onset of swap at about 1.2GB of the transfer...

Varied from 41.6 down to 17.5 then rose to 39 for several instances(6) and ended at 22 to 25MB/S. RAM free=54MB.

2.023856 in use
Loaded 5.9GB .iso at steady 27MB/S....

Loaded videos 5GB at nominal rate of 25 to 27MB/S..

Loaded Packages .tsr.gz At nominal 24 to 27MB/S

Total swap size = 19.7 GB

Entering this post from that system loading.

Response is a little slower but acceptable when addressing the internet.

Next is to see what happens in Larch boot to ram.....

It seems I take a hit with 3GB in this mobo....

lilsirecho · 2006-10-13 22:48:26

In larch copy-to-ram with 2GB of RAM in the machine:

Initial transfer of 3.6GB of mp3 material varied from the initial swap start point of around 1.2GB :

Ran below 20 for 35 % of transfer then rose to an average of 22MB/S for the remainder of the 3.6GB file. (mixed data and mp3 songs)

Next was .iso for larch of 5.9GB : average speed 23MB/S for 5.9GB

Next was 5+ GB of video files averaged 20+ MB/S....

Final was packages .tar.gz 5.9GB averaged 20MB/S

Ram used 2.012824KB 65MB free Swap20GB 18GB free

Next test @ 1GB in Larch copy-to-ram...

lilsirecho · 2006-10-13 23:38:28

In larch copy-to-ram with 1GB of ram (~700MB of larch boot).

NOTE: last test ram free increased to 470MB after hda3 was unmounted.

Mp3 and data file 3.6GB began swap earlier than 1GB and was slow to reach 20MB/S but at 65+% it carried on at 22MB/S average

The 5.9GB .iso averaged 21MB/S......

The video files Loaded at 19+ MB/S steadily......

Packages of 5.9GB averaged 19MB/S with variations above 22 and below 17.......

Ram free at the moment is 22MB with hda3 mounted, unmounted it is 182MB.

Swapoff -a reduces swap to zero and the ram free increases to 332MB.

Tmpfs allocated to larch is 607MB with this larch boot @ 1GB ram.

One last test with the larch boot at 1GB with USB swap drive WD400 HD as USB swap at 40GB.

lilsirecho · 2006-10-14 00:20:56

The transfer test of Larch with 1GB of ram using external USB swap of 40GB produced a transfer rate of ~14.5MB/S for all transfers.

I will mount a parallel Maxtor with 20GB of swap on the USB bus and do a final test.

My larch boot is greater than the allocated tmpfs 60% of RAM.

lilsirecho · 2006-10-14 01:39:08

Testa with two swap drives in USB priority 3 resulted in approximately 16.5 to 17 MB/S performance throughout with some variations in the mp3 file.

Again, the tmpfs larch size is 607MB, some 100MB less than the larch .iso.

Doubling of the transfer rate did not occur. The single drive wd400 performs at nearly the rate of both drives combined.

It may be that I need to provide more tmpfs size for larch.

End of tests for now.

lilsirecho · 2006-10-14 03:30:30

Repeated the test of wd400 swap drive as hdb and ran /mytmpfs in arch boot.

The results confirm the original transfer speeds with 1GB of RAM. The hda3 was addressed for the transfers.

The speed of transfer to wd400 in USB external 2.0 is ~one half of the speed inobtained in the internal slot test.

I may try to increase the tmpfs RAM allotment to 90% in a new larch DVD and see what gives.

I plan to try another manufacturer for the parallel drive test but have to forego it until I can borrow same.

The tmpfs of Larch is set at 60% of ram. The ram free in 1GB ram was ~12, in 2GB ram it was ~52 and with 3GB ram ram free was ~86 during the time hda3 was mounted and transfers being made. These figures may relate to the kernel kswapd limits managed in the OS.

The wd400 drive seems to perform better than the maxtor but haven't tried the maxtor in the internal slot as yet for comparison. It is programmed with 20GB of swap as partition2.

Guess that is next....

lilsirecho · 2006-10-14 06:05:50

Results of tests with internal swap drive paralleled with external swap drive.

Tests indicate that a proximity of swap to the machine permits much higher swap speed in transfers.

With the wd400 in hdb1 swap and the maxtor 20GB sda2 swap, the transfers in Larch copy-to-ram were always averaging above 23MB/S.

Utilizing a second external USB/IDE assy for the maxtor, permitted the transfers at ~29MB/sec average.

The exact nature of the swap operations are unpredictable altho one might conjecture that the demands of the system are better served by kswapd with an internal swap drive. The striping with paralleled swap drives is better served with the internal drive and the swap transfer speed holds steady right up to the maximum programmed swap size.

I have one more option to investigate since my system has raid primary and secondary connectors. It may be that the internal swap could be provided by the raid primary wd400 drive and relieve the primary/slave interface as utilized in these tests.

The ram free varied from ~20 to 62MB in this test of dual internal/external swap drives. The external drive was active at all times.

I suspect it slows things down some!!!!

I do not have a drive in arch that will provide swap of 40GB in its partitions. I guess I will have to generate one.....

lilsirecho · 2006-10-15 17:24:50

Up to this point I have tried many times to increase swap transfer speed in USB 2.0 config as well as internal drive transfers by parallel priority methods.

I have noticed two to five MB of difference in a few cases but never a consistent doubling of swap performance.

In googling, one recommend was to partition a single drive with multiple swaps each with the same priority. This being done with 10G per partition reduced the swap transfer speed from 30MB/S to 10MB/S in one trial recently completed. Thus, it certainly indicates a single partition of large size is more efficient.

In utilizing swap extension of RAM as I am attempting, the usual implementation of swap is probably not applicable. Seemingly, it applies to smaller partition sizes and certainly is impacted by content.

I now have 3GB of RAM installed and obtain 35 to 40MB/S transfer speed with the wd400 drive in hdb slot while utilizing the full drive in arch boot. This drive is dma 100 while the maxtor in hda is dma 133. This reduces the performance of the hda drive somewhat. The transfer rates encountered reached as high as 43MB/S as indicated during transfers

Paralleling this wd400 with a maxtor drive in secondary master did not make a significant swap transfer speed change, the maxtor having a 16GB swap partion in hdc2.

A final exam might utilize a dma 133 drive in hdb to compare with the dma100 utilized for my tests.

While in Larch boot, hda3 was mounted to make transfers for the tests. This is a large impact on RAM in use. I plan to try mount of the larch DVD, which has the 5.9GB of packages .tar.gz, and test the transfer from that source. That should reduce the ram in use and minimize swap maintenance of system loads.

The speeds obtained while swap is being utilized are comparable to the speeds provided by the transfer with ram support up to the point when swap takes over. Thus, it suggests that no further performance would be possible except the dma133 change to hdb, since it is limiting the hda3 dma133 to 100dma.

Comparing the ram performance before swap is paramount seems to be a poor comparison since swap is performed with ram operative in creating the "paging" necessary for swap transfer.

However, in view of the transfer speeds obtained with swap, the ram functions required seem not to slow the transfers when compared to the speed with ram operating within the 3GB ram size limit, in my system.. It acts similar to a normal transfer within the system.

Perhaps the hda drive should be programmed with a large swap partition, thus limiting the computer to one drive required and extended ram would be possible with dma 133 and provide max transfer rate.

External USB rates reached a max of 17MB/S with hard drives paralleled swap and ran 15MB/S with single drive swap. The performance is consistent with normal transfers from external USB drives in ext3 partitioned files...USB limited.

I have not spent much time in operating the system with demands from the extended swap data, but do note that unmounting the /mytmpfs directory leaves about 500MB of swap to be deactivated, indicating that the impact on the system may not be very significant since the system has 60GB installed in hda3, much of which is "least required".

Utilizing the packages tar.gz installed with the DVD mounted after booting Larch indicates no impact on pacman transfers from that directory, the install of openoffice-base taking 27 secs.

Onward!!!

lilsirecho · 2006-10-15 18:28:08

As expected, transferring the 5.9GB packages directory into /mytmpfs while in Larch produced a transfer rate limited by the DVD dual-layer Sony DVD read speed but was a steady 8 to 12MB/S rate while swap transfer was progressing throughout the 5.9 GB transfer.

I expect an improvement in transfer speed while utilizing a dma 133 drive with a large swap partition. perhaps a steady 45mb/s for most data.

That will be my last experiment ......

T-Dawg · 2006-10-15 18:31:29

who are you talking to?

shadowhand · 2006-10-15 18:59:44

Penguin wrote:

who are you talking to?

Better question: What are you trying to prove here? No one uses a swap partition greater than 4xRAM, and I never seen it recommended to use more than 2Gb swap, so..... what did you prove?

lilsirecho · 2006-10-15 19:12:14

One last note:

Commands useful in addition to the swap commands previously listed include;

TOP ....lists ram use, swap use and system activities.

SWAPON -s .... lists swap partitions active.

I apologize for not having a system running without GUI which might well improve the operations tested.

Hoping these tests provide useful info for users..

EDIT: I assume DMA has no meaning with USB 2.0 mounted drives with IDE interface devices since direct memory access is not viable. What this means for a given drive is probably the higher the drive rotation speed the better the transfer but still limited by USB 2.0 interface limits.

Ray

lilsirecho · 2006-10-15 23:25:05

I didn't [prove anything, just experimented.

I haven't yet found a limit to swap partition size in spite of directives to the contrary.

I did show that packages installed in a DVD dual-layer Larch would be accessed faster than internet pacman calls and provide local access to all included packages on the dvd.

I did show that USB 2.0 permits use of external swap drives at ~15MB/sec transfer speeds in extended ram . That is about the usual 2.0 USB transfer speed. Thereby, one can install via Larch a very large package directory, while booting a normal 700MB arch OS, and load all in copy-to-ram. Whatever advantage there is in copy-to-ram is then evident.

If, in future days, USB 2.0 is improved, great! If in future days, flash drives are less fragile, great! Then the power needed to do what the drives do now will be greatly reduced and also removed from the computer case.

I still don't understand the rationale behind the limits expressed for swap in view of my many runs utilizing many GB of swap. Only in tmpfs is extended ram useable and I spent many hours extending same to test experimentally what limits are to be found.

I am surmising that swap limits as defined in Linux are esoteric and have caveats not expressed. The mechanisms involved leave one confused.javascript:emoticon(':?')

No offense gents, but an effort to generate ARCH DISCUSSION which is what my experiment was all about in this forum.

Most users are interested in running smaller. leaner arch OS boots and this discussion forum subject may not interest many.

I do feel that a live DVD boot to ram with 3200 packages included and extended ram swap is worth some discussion. However, I could be way off base!!!! The 3200 packages is extreme but does prove(!) that these can be accessed in copy-to-ram and installed therein to customize a live larch copy-to-ram arch (or any other OS).

I have a quote from googling:

=======================================================================
The 2G limit never existed, it was an artificial limit in the swapon
utility, each arch has its own limited mandated by the pagetable layout
and it's usually way above 2G and a non pratical one normally
===================================================================================
Other googles say parallel swap partitions (on separate drives or the same drives) can be utilized to increase swap speeds by setting equal priority for all. I haven't found the secret to this setup.

I think that's a discussion item, for sure.

I may have proved that no one is interested!!!!

Lone_Wolf · 2006-10-15 23:34:08

Not sure what practical uses this has, but it certainly has been an interesting read.

lilsirecho · 2006-10-16 00:15:49

One factor involved is the usual statement " swap is hundreds of times slower than ram".

If one desires to utilize greater than ram limits, the swap extended ram is utilized in my application of Larch copy-to-ram. When so used, the disc containing the data contained in the larch boot( added packages for example in my case) can be re-inserted into the rom device and will load data at speeds that are not hundreds of times slower than real world useage of downloads via internet. Only the install portion of downloads is required, rapid!

I suspect that the pacman Ud (package name) produces at least 10MB/s transfer of packages from the reinserted dual-layer DVD I utilized. This eliminates the internet delays inherent with pacman downloads.

I point out that all elements within the system are mounted in ram ( and extended ram). I have found no limit to this except the DVD dual-layer size. The USB drive used in extended ram is limited to ~15MB/S and this is a practical real-world figure (not a specmanship).

I have demonstrated the experimental results using 1G to 3G of ram with no effect on system performance. This is real world data.

If the user wants an upgrade, pacman will provide same for the duration of the boot-up.

An experiment with capability to extend live copy-to-ram arch custom boot capability with external extended ram swap drive(s).

Testing included trials of parallel real-world priority controlled swap partitions...none of which provided any improvement.

However, with everything in ram, except the external swap drive, USB 2.0 limits the speed of transfers (real world again).

In my system, I have transferred large files from hdb to hdc at 50mb/s, which is an indication of real world capability.

Pacman downloads normal speeds are 88 kilobits/second with my DSL.

Anyhow, I still have hopes for better flash drives in orderto reduce the power needed to run the computer by utilizing them externally as swap devices to extend ram as well as read-only for data devices.

Maybe its a dumb idea?

EDIT: One factor not mentioned is that the Live Larch boot is a testing ground not likely to mash the arch system on your main drive, in fact, it can be shut down to ensure same! The cd/DVD rom can enter data till the cows come home if the external swap drive has terabyte capability. Also, entries fromUSB devices.........

lilsirecho · 2006-10-16 23:51:30

Tried one more time to get a result with paired swap drives.

I used the maxtor with 16+GB of swap in hdb.

I used the wd400 drive as sda1 with 40GB of swap.

Loaded the system to 25GB of swap files running well past the 16GB of swap in hdb drive.

The USB sda1 drive indicated activity at onset of swap transfers and the results were indicative of parallel swap function.

Throughout the swap loading the speeds were 25MB/S to 35MB/S.

This indicates that even the 16GB drive was not fully loaded when the /mytmpfs limit of 25GB was reached verifying that parallel swap operation was occuring in addition to the fact that the external drive was showing continuous activity.

Entry of data via flash drive with 2.0GB device occured at 17MB/S , ext3, into /mytmpfs directory.

lilsirecho · 2006-10-18 02:56:51

Further data from tests with large swap.

Installed maxtor DMA133 in both hdb and sda1. A 20GB partition in hdb2 and the 16GB partition of sda2.

Programmed 70G for /mytmpfs and transferred data files up to 36,374,892 KB with 3,578,740 KB free. The USB drive was active from the onset of swap with my 3GB ram system which occured at 1.5GB of swap in use.

The transfers of .tar.gz packages was repeated with renaming each transfer for three such transfers of 5.9GB each. All transferred with sda1 active.

Initially, over 40MB/sec occured , as high as 43, and the remaining ~4.5 GB transferred at 35MB/S average. This pattern repeated for all three renamed packages transferred.

Several .vob files were transferred at rates averaging 25 to 30MB/S. The .iso file of 5.9GB averages ~33MB/sec.

All files thusly available in swap in /mytmpfs access directory were then copied into home directory to examine the transfer speeds out of swap.

The sda1 USB swap drive showed activity throughout and essentially 95% of all files of the size 1GB to 5.9GB transferred at an average of 5 to 7 MB/S. This indicates identical speeds across the full drive partition, suggesting the bottleneck is within the USB/swap management arena.

Comparing these speeds to that of a flash drive having 15-17MB/S performance in transfers into the computer, indicates that improvement is sorely desired.

The transfer speeds into the swap drives indicates a sharing of swap transfers which exceeds the transfer speeds of USB swap drives utilized alone by a good margin

The partitions used are by no means a usual system size, however, each of the two drives used in this test have windows partitions in the first partition which may have a bearing on the transfer speed out of the swap partition(s).

lilsirecho · 2006-10-18 18:31:14

Made some changesto the method of loadinginto tmpfs with swap.

Googled extensively and learned.

To this point, have used separate tmpfs directory to enable much experimenting without interfering with the normal tmpfs mount while in copy-to-ram in Larch.

The following used maxtor 20G in hdb2and wd400 in sda1 40G.

The tmpfs file has been remounted to 40G size for these tests as follows:

#mount -o remount,size=40G tmpfs

I then mounted a version of Larch DVD +rw which had two items included, the mp3 downloads(3.6G) and a 1G .avi file.

The 3.6G file loaded into tmpfs at speeds varying from12 down to 7MB/S, mostly 7 to 9. The .avi file speed was a steady 14-15MB/S.

I then loaded tmpfs from mounted hda3 source(maxtor 80GB DMA133) the larch .iso file which loaded at an aveage speed of 22MB/S (5.9G).

I then loaded five .vob files all of them entering tmpfs at speeds 20 to 24 MB/S.

Last transfers were from packages .tar.gz (5.9G) , two instances, these loaded at speeds averaging 23 MB/S.

I expect an increase in speeds with a full-size 133 swap drive in hdb and a similar drive USB slot.

The cpu use during transfers was maximum and many "stalled" indications occured while transferring. The system loads are greater with Kde operative and speeds would be affected. These figures are real-world GUI.

Useful commands also include df and du.....

Tmpfs is now loaded with 23.6GB of swap, basic ram 3GB 100MB free with hda3 mounted. Unmounting hda3 provides an indicated 533MB free. Tmpfs is 60% full(40G enabled).

Mplayer was downloaded via pacman and entered the swap environment. It performs well in running videos AFAICT, and the mp3's also run OK in mplayer.

Just now installed openoffice-base from tmpfs packages via #pacman -Ud (paste openoffice-base tar.gz) and it loaded to root prompt in 50 seconds. It doesn't show in Kde which may need an upgrade to indicate it's presence.

Locate command not useable, running just now FIND FILES reports 8 /opt files read-write whereupon I closed the search.

I suspect I can open the office elements from /opt later on.

A side note, mplayer is included in kde multimedia menu.

Googling revealed that others use tmpfs in the manner I am investigating so I am not the only "wild and crazy guy" if such be true!!!!

I am presently in Larch copy-to-ram with all of the swap extended ram in place and operating as needed.

Loading via the DVDrom with DVD +RDL provided an acceptable 15MB/S speed for video file, perhaps because the data was located past the 4GB level on the disc. It may be that placing this data (or all desired data) near the 6-8GB level on the DL disc since the indications during transfer showed a steady increase of 0.1 MB/S as the transfer progressed. Projecting this speed to the higher level disc position might be sufficient to double the transfer rate for those files near the 8GB limit. Thus DVD entry of data would be enhanced.

A single HDD fully swap , coupled with an external USB 2.0 swap drive, seems to be a useable arrangement. The system would have one drive running internally and the DVDrom. If external swap is sufficient, no drive would be energized internally except perhaps the DVD. External USB flash drives produce 15MB/S transfers as well.

Now maybejavascript:emoticon(':lol:') no mobo?

benplaut · 2006-10-18 18:36:39

.

Last edited by benplaut (2021-06-25 12:39:05)

lilsirecho · 2006-10-18 19:32:03

Loaded via pacman , OPERA, in 2 secs. Writing this in that browser.

lilsirecho · 2006-10-18 19:44:35

Mozilla-firefox loaded from pacman -Ud in five secs...it has 53 dependencies.

lilsirecho · 2006-10-18 19:59:56

benplaut;
Look in TOP, DU or qtparted for swap details. If in KDE, sys guard lists the swap size as well as the ram, both in use and free.

lilsirecho · 2006-10-18 20:30:32

[url]http://www.linux.or.jp/JF/JFdocs/kernel … /tmpfs.txt

The above url has a good discussion of applying tmpfs for large swap ram extension.[/url]

lilsirecho · 2006-10-18 23:49:16

Latest change involves placing the swap on the "tail end".

Re-partitioned wd400 external drive to ext3 for one half and swap for the second partition.

Retried the transfers and found a consistent 25MB/S transfer rate for all types of data involved in those same packages I have benn running. Loaded from desktop in arch for this test using mounted /.mytmpfs.

Transfers from Hdc DVD rom using DVD+rwDL gave results consistent with the previous runs....7MB/S to almost eleven. Transfer of Kernel26beyond took one second.

While performing this test, no "stalled" incidents occured.

I will try this again in Larch c2r......

Arch Linux

#26 2006-10-13 21:27:28

Re: Extended memory with swap in ARCH (swap to 30GB)

#27 2006-10-13 22:09:24

Re: Extended memory with swap in ARCH (swap to 30GB)

#28 2006-10-13 22:48:26

Re: Extended memory with swap in ARCH (swap to 30GB)

#29 2006-10-13 23:38:28

Re: Extended memory with swap in ARCH (swap to 30GB)

#30 2006-10-14 00:20:56

Re: Extended memory with swap in ARCH (swap to 30GB)

#31 2006-10-14 01:39:08

Re: Extended memory with swap in ARCH (swap to 30GB)

#32 2006-10-14 03:30:30

Re: Extended memory with swap in ARCH (swap to 30GB)

#33 2006-10-14 06:05:50

Re: Extended memory with swap in ARCH (swap to 30GB)

#34 2006-10-15 17:24:50

Re: Extended memory with swap in ARCH (swap to 30GB)

#35 2006-10-15 18:28:08

Re: Extended memory with swap in ARCH (swap to 30GB)

#36 2006-10-15 18:31:29

Re: Extended memory with swap in ARCH (swap to 30GB)

#37 2006-10-15 18:59:44

Re: Extended memory with swap in ARCH (swap to 30GB)

#38 2006-10-15 19:12:14

Re: Extended memory with swap in ARCH (swap to 30GB)

#39 2006-10-15 23:25:05

Re: Extended memory with swap in ARCH (swap to 30GB)

#40 2006-10-15 23:34:08

Re: Extended memory with swap in ARCH (swap to 30GB)

#41 2006-10-16 00:15:49

Re: Extended memory with swap in ARCH (swap to 30GB)

#42 2006-10-16 23:51:30

Re: Extended memory with swap in ARCH (swap to 30GB)

#43 2006-10-18 02:56:51

Re: Extended memory with swap in ARCH (swap to 30GB)

#44 2006-10-18 18:31:14

Re: Extended memory with swap in ARCH (swap to 30GB)

#45 2006-10-18 18:36:39

Re: Extended memory with swap in ARCH (swap to 30GB)

#46 2006-10-18 19:32:03

Re: Extended memory with swap in ARCH (swap to 30GB)

#47 2006-10-18 19:44:35

Re: Extended memory with swap in ARCH (swap to 30GB)

#48 2006-10-18 19:59:56

Re: Extended memory with swap in ARCH (swap to 30GB)

#49 2006-10-18 20:30:32

Re: Extended memory with swap in ARCH (swap to 30GB)

#50 2006-10-18 23:49:16

Re: Extended memory with swap in ARCH (swap to 30GB)

Board footer