Scalability Testing with NFS File-IO

First, 500 MAC-VLANs will need to be created.
1. In the Port-Mgr tab select port eth1 and click Create.
  1. Select MAC-VLAN.
  2. Set Quantity to 500.
  3. The starting IP address is 10.17.1.2/16 for this test. If your file server is on a different network, change the IP here accordingly. Make sure not to use the file server's IP address.
  4. Click Apply and close the Port Create window once all ports are configured. Make sure all MAC-VLANs get IPs, this may take some time.
Create 250 NFS writers and 250 NFS readers. Batch create 50 at a time (49 for first batch of reader/writers). More information on creating File-IO endpoints can be found here: LANforge File-IO with CIFS and NFS
1. Any non-default settings for the NFS writers are listed below.
  1. Name is nfs-writer001.
  2. FS-Type: NFS.
  3. Port is eth1#0.
  4. Min-RW and Max-RW sizes are 4k.
  5. Min/Max File sizes are 1MB.
  6. Min/Max Write rates are 100Mbps.
  7. Quiesce After is Forever.
  8. Server is 10.17.1.1:/mnt/tmpfs1 (use your own file server settings here).
  9. Make sure Read/Write is set to Write.
  10. Turn on the Use O_DIRECT checkbox.
2. Any non-default settings for the NFS readers are listed below.
  1. Name is nfs-reader001.
  2. FS-Type: NFS.
  3. Port is eth1#250.
  4. Min-RW and Max-RW sizes are 4k.
  5. Min/Max File sizes are 1MB.
  6. Min/Max Write rates are 100Mbps.
  7. Quiesce After is Forever.
  8. Server is 10.17.1.1:/mnt/tmpfs1 (use your own file server settings here).
  9. Set Read/Write to Read.
  10. Set Prefix to nfs-writer001.
  11. Turn on the Use O_DIRECT checkbox.
Start the first 50 writers then the first 50 readers. It's recommended to slowly start File-IO endpoints, for example, starting 25 writers then 25 readers then the same again until you have the desired amount running. The reason to do this is because the system may become unresponsive if it can't handle the number of File-IO endpoints.
Note: If you notice the system slowing down, connections not starting, or connections stuck in WAITING state, or any weirdness in general, you should stop 25 or 50 writers/readers at a time until the system clears up.
Now that 50 File-IO endpoints are running, the memory will be analyzed via htop.
1. Open a terminal on the LANforge system using a preferred method (either directly or remotely through ssh, rdesktop, or vncviewer).
2. Show a list of btserver processes (LANforge processes) by running the command:
  htop -p `pgrep btserver | tr "\n" "," | sed 's/,$//'`
  If htop isn't installed run: sudo yum install htop
  1. The majority of btserver processes with the same or similar values here represent the running File-IO endpoints. The RES (resident value) column represents the memory used by these File-IO endpoints. The current test uses an average of about 10,400 KiB or 10.6 MB per btserver process. These values should be monitored as the File-IO settings are adjusted.
    
    NOTE: You will need to restart htop if the endpoints are stopped/restarted. Just press q to quit out and run the above htop command again.
3. Total up the RES memory used by btserver. This is useful to compare memory usage between a different number of connections. For example you could compare how much memory 100 endpoints use compared to 500 endpoints.
  1. To show the total RES memory used, run the command:
    ps aux | grep btserver | awk '{s+=$6} END {print s}'
    To repeat it every second, use the below command. Press Ctrl+C to stop.
    while sleep 1; do ps aux | grep btserver | awk '{s+=$6} END {print s}'; done
  2. The total memory for 50 NFS File-IO endpoints with a Min/Max RW rate of 4k is about 1,130,000 KiB which converts to around 1,157 MB.
While the File-IO endpoints are running, change the Min-RW and Max-RW settings.
1. Select all File-IO endpoints and click Batch Modify.
  1. Set the Min/Max RW sizes to 1MB.
  2. Click OK
Analyze how the Min/Max RW size setting change has affected the memory (see steps 4B and 4C).
1. Changing the Min/Max RW sizes to 1MB increased the btserver memory use to about 11,600 KiB or 11.9 MB from 10,400 KiB or 10.6 MB. About a 1,200 KiB or 1.3 MB difference.
  
  The total memory increased to 1,233,000 KiB, 1,262 MB from 1,130,000 KiB, 1,157 MB. About a 103,000 KiB or 105 MB difference.
Set the endpoints Min/Max RW size back to 4k.
1. Stop all running file-IO endpoints by selecting them and clicking Stop.
2. Select all file-IO endpoints and click Batch Modify.
  1. Set the Min/Max RW sizes to 4k.
  2. Click OK.
Repeat steps 3 through 7 for 125 writers/readers and then again for 250 writers/readers.
The results from the above tests are shown in the below tables. They show how RW size affects 100, 250, and 500 NFS File-IO endpoints.

		sales@candelatech.com support@candelatech.com +1 (360) 380-1618 [PST, GMT -8]
	Network Testing and Emulation Solutions

Scalability Testing with NFS File-IO

Goal: Analyze memory usage as the number of NFS clients is scaled up, and also when NFS endpoint settings are tweaked.