Kruskal Cluster

Martin David Kruskal



The kruskal cluster consists of 32 HP DL165G7 servers each with dual 16 core AMD Opteron processors (total 1024 cores). They are interconnected by Infiniband Quad Data Rate (40Gb) switches. These systems each boot from the same Red Hat Enterprise Linux Operating System image.

The kruskal queue schedules batch jobs on the kruskal cluster. The cluster is optimized for mid scale (>=32 core) parallel jobs, and is not intended to run small jobs which can be accommodated by other clusters.

Tips To Run Kruskal Jobs

1. Users are encouraged to build their executables with "pathscale/3.2" and "openmpi". Applications built with MPICH will run on kruskal through IPoIB, but this won't be able to fully utilize underlying Infiniband capacity through RDMA.

2. OpenMPI is integrated with the batch system. It uses a considerable amount of shared memory with RDMA technology. Users running openmpi need to increase this limit:

for a C shell user, put the following line in ./cshrc file:

limit memorylocked 1048576

for a Bash shell user, put the following line in ./bashrc file:

ulimit -l 1048576

3. OpenMPI attemps to determine which BTL driver to use at run time. The following example shows how to use the mca option (--mca) to instruct it to use InfiniBand driver: ("np" is number of cores)

mpirun --mca btl openib -np 128 myprog

Note: This is only recommended for older versions of openmpi, such as version 1.2.7. "--mca btl openib" became obsolete in version 1.3 and up. You do not need mca option anymore. Adding it improperly could stop your program from running.

4. Here is a sample job script:

      ## --- run on the kruskal queue
      #PBS -q kruskal
      ## --- the name of the job
      #PBS -N my_job
      ## --- get 8 systems, 16 cores on each, and 32GB of total memory
      #PBS -l nodes=8:ppn=16,mem=32000mb
      ## --- don't rerun if the job fails
      #PBS -r n
      ## --- the time needed
      #PBS -l walltime=24:00:00
      cd $PBS_O_WORKDIR
      mpirun --mca btl openib -np 128 ./prog