InfiniBand Network Information

Last Modified: 05/04 16:57

USF provides access to 240 processor cores connected by 4x Single-Data-Rate (SDR) InfiniBand and over 1200 processors connected by 4x Double-Data-Rate (DDR) InfiniBand.

SDR Information

SDR InfiniBand has a peak theoretical bandwidth of 10Gbps, minus 20% for protocol overhead. This provides a peak usable bandwidth of roughly 1GB/s. Latencies across this fabric are roughly 3 microseconds, point-to-point for smaller message sizes. The bandwidth and low-latency provided by this fabric is generally ample for most parallel workloads.

DDR Information

DDR InfiniBand has a peak theoretical bandwidth of 20Gbps, minus 20% for protocol overhead. This provides a peak usable bandwidth of roughly 2GB/s. Latencies across this fabric range from 1.3 to 2 microseconds point-to-point for smaller message sizes. The bandwidth and low-latency provided is quite ample for most parallel workloads.

Using InfiniBand

To specify the use of InfiniBand by your job, regardless of SDR/DDR capabilities, include the following in your job script:

#$ -l i_ib=true

This will ensure that your job is placed on nodes in either the SDR or DDR networks.

To specify the use of DDR InfiniBand, include the following option instead:

#$ -l i_ib_ddr=true

and for SDR InfiniBand:

#$ -l i_ib_sdr=true

Low-Priority Queues

For users who are not a part of research groups that own InfiniBand hardware and wish to utilize more processors on a Low-Priority basis (see GridEngine Dispatch Policy for more information on Low-Priority queues), specify the following complexes to get low-priority access to InfiniBand connected hosts:

#$ -l p_low=true,i_ib=true

You can also specify the SDR/DDR attributes as well, if you wish to be more specific about the type of fabric your job runs on.