PPPL Linux Users Guide

Contents

  1. Introduction
  2. Available Hosts
  3. Logging In
  4. Accessing and Storing Data
  5. Finding Programs
  6. Using Modules
  7. Login Scripts
  8. Tips on Compiling Code
  9. Submitting Jobs
  10. Printing
  11. Getting Help

Introduction

This document is geared towards helping users with some Linux or UNIX experience learn the details of the Linux systems at PPPL specific to our site. If you're new to the Linux world, please take a look at http://www.tldp.org/LDP/intro-linux/html/ first for a tutorial on the basics, before continuing on with this guide. For the more seasoned Linux user, please be sure to read through this guide, as it contains many tips that will help you work efficiently on PPPL's systems.

Available Hosts

The main entry point to the PPPL Linux environment is a host called portal.pppl.gov.   Portal is not an actual host, but rather an alias to a pool of nodes available for general login sessions. The name portal will automatically adjust itself to point to the best (least loaded) system in the interactive portal pool. The portal nodes all have publicly accessible IP addresses; you will be to access them directly from just about anywhere assuming that you've authenticated with PPPL's firewall first.

So you use portal as your entry point to PPPL's Linux systems.

There are many other Linux hosts contained in clusters, and not for direct logins. These hosts are reserved by submitting a job to a job scheduler. The job scheduler will determine what hosts you need for your job, then, once the hosts are available, start the job. See hardware for information about the systems available for running jobs.

All nodes at PPPL run a 64 bit version of Linux based on RedHat Enterprise Linux V 5, including all of the general use nodes in portal. There is a small group of nodes running the previous version of Linux in a group called portalr4, which is a pool of general use machines just like portal. To access these previous version systems, ssh to portalr4.pppl.gov.

Logging In

All Linux hosts running RedHat Enterprise Linux can be accessed remotely via SSH. SSH is the only mechanism for accessing these nodes - other protocols such as Telnet, RSH, REXEC, RCP, FTP, X, XDM, and XDMCP are not available due to security concerns. You will however, be able to run X applications using X forwarding included with SSH. All users are allowed to login to the portal nodes via SSH at any time. Even though the other Linux hosts have SSH enabled, users are not allowed to login directly, unless that user reserved the node via the job scheduler. Once a node is reserved with the job scheduler, that user will automatically be granted access to directly login to a computational node for the duration of their job.

Accessing and Storing Data

Almost all the file systems on the PPPL Linux systems are served from a central server. Here is list of what file systems are available, what they hold, and who can access them:

Finding Programs

On the PPPL Linux systems, programs can be divided into two categories: those provided as part of Linux, and those installed by PPPL. Programs included with the system, such as awk, grep, sed, vi, bash, gcc, autoconf, and xterm, are installed in the standard location, usually /bin or /usr/bin. Likewise, the basic system libraries and headers are found in places like /lib and /usr/include.

Programs installed by PPPL administrators, such as code compilers, IDL, MPICH, and MDSplus clients are found in /usr/pppl. These programs are installed in directories beneath /usr/pppl, dependent on the package name and the version. For example, version 6.20c of Lahey-Fujitsu Fortran can be found in /usr/pppl/lff95/6.20c. This allows PPPL to maintain different versions of packages at the same time. The version specific installation path paradigm is taken even further in the case of libraries that depend on a specific compiler. For example, if one wanted to use NetCDF, one would find the NetCDF installation directory within the installation folder of the compiler used to build NetCDF. NetCDF 3.6.0 for Lahey-Fujitsu Fortran can be found in /usr/pppl/lff95/6.20c/netcdf-3.6.0 and NetCDF for the PathScale v2.0-197.2758 compiler is found in /usr/pppl/pathscale/2.0-197.2758/netcdf-3.6.0. This scheme allows both administrators and users to avoid mixing different versions of a package - i.e. it becomes very difficult to get the man packages for one version of IDL and the binaries from a different version.

In order to help users navigate around /usr/pppl, we've setup a package called Modules.

Using Modules

As was mentioned in the section above, /usr/pppl has a unique path for every piece of software installed by PPPL. In order to make it easy to manage all the different paths that software can be found in, PPPL has setup a package used at many supercomputing sites called Modules. Modules does what the setup scripts (such as those included with IDL or Lahey-Fujitsu Fortran) did - it sets environment variables, updates your PATH, and sets aliases in your shell. Modules also has an 'unload' capability; after you're done using a package, you can return your environment to its previous state. This feature is important if you wish to switch from one version of a package to another.

Here is a tutorial on how to use Modules: Then first command to learn is 'module avail' - this will show you a list of packages that are currently available for you to use:

	[me@portal ~]$ module avail
	
	----------------------------------- /usr/pppl/Modules/modulefiles ---------------------------------
	freetds/0.62.4(default)         lors/0.82(default)              totalview/6.4.0-0
	globus/2.4.3(default)           mdsplus/grid                    totalview/6.5.0-0
	globus/gt3.2.1                  mdsplus/gt3.2.1                 totalview/6.5.0-2
	hdf-view/2.0(default)           mdsplus/nongrid(default)        totalview/6.6.0-0
	ibp/1.3.1.2(default)            pathscale/2.0-197.2758(default) totalview/6.6.0-2(default)
	idl/6.0(default)                pdsh/2.3-1(default)             use.own
	iperf/1.7.0(default)            perl/5.8.5                      valgrind/2.2.0(default)
	java/1.4.2_05(default)          perl/5.8.6(default)             yorick/1.5.14(default)
	lff95/6.20c(default)            pgf/5.2-1(default)
	libxio/0.1(default)             pvfs2/1.0.1(default)
The next command to learn is 'module list'. This will show you what modules you currently have loaded.
	[me@portal ~]$ module list
	No Modulefiles Currently Loaded.
As you can see, no modules are currently loaded in my environment. To load module, such as perl, I can simply issue the following command:
	[me@portal ~]$ module load perl
	[me@portal ~]$ module list
	Currently Loaded Modulefiles:
	  1) perl/5.8.6
	[me@portal ~]$ which perl
	/usr/pppl/perl/5.8.6/bin/perl
The default version of perl, 5.8.6, was loaded into my environment by simply loading 'perl'. If I specifically, wanted to have version 5.8.5, instead of the default version 5.8.6 perl, I could have issued 'module load perl/5.8.5'. Modules also have conflicts; for example I can only have one version of perl loaded at a time. If I already have loaded the default (5.8.6) version of perl, and I then try to load version 5.8.5 of perl, I will get the following error:
	[me@portal ~]$ module list
	Currently Loaded Modulefiles:
	  1) perl/5.8.6
	[me@portal ~]$ module load perl/5.8.5
	perl/5.8.5(8):ERROR:150: Module 'perl/5.8.5' conflicts with the currently loaded module(s) 'perl/5.8.6'
	perl/5.8.5(8):ERROR:102: Tcl command execution failed: conflict perl
In order to switch to using another version of perl, I can use 'module switch':
	[me@portal ~]$ module list
	Currently Loaded Modulefiles:
	  1) perl/5.8.6
	[me@portal ~]$ module switch perl perl/5.8.5
	me@portal ~]$ module list
	Currently Loaded Modulefiles:
	  1) perl/5.8.5
	[me@portal ~]$ which perl
	/usr/pppl/perl/5.8.5/bin/perl
To unload a module, I can use 'module unload':
	[me@portal ~]$ module unload perl
Finally, if I wish to remove all modules, I issue 'module purge':
	[me@portal ~]$ module purge

A special case with Modules is the way that compiler dependant libraries are handled. Things like MPICH and NetCDF are built with a certain compiler, and require that compiler to function properly. Because of this relationship, the compiler dependant functions do not show up on the initial list of available modules. However, once you load the module for a compiler, an additional subset of modules specifically for that compiler will appear:

	[me@portal ~]$ module avail
	
	----------------------------------- /usr/pppl/Modules/modulefiles ---------------------------------
	freetds/0.62.4(default)         lors/0.82(default)              totalview/6.4.0-0
	globus/2.4.3(default)           mdsplus/grid                    totalview/6.5.0-0
	globus/gt3.2.1                  mdsplus/gt3.2.1                 totalview/6.5.0-2
	hdf-view/2.0(default)           mdsplus/nongrid(default)        totalview/6.6.0-0
	ibp/1.3.1.2(default)            pathscale/2.0-197.2758(default) totalview/6.6.0-2(default)
	idl/6.0(default)                pdsh/2.3-1(default)             use.own
	iperf/1.7.0(default)            perl/5.8.5                      valgrind/2.2.0(default)
	java/1.4.2_05(default)          perl/5.8.6(default)             yorick/1.5.14(default)
	lff95/6.20c(default)            pgf/5.2-1(default)
	libxio/0.1(default)             pvfs2/1.0.1(default)
	[me@portal ~]$ module load lff95
	[me@portal ~]$ module avail
	
	----------------------------------- /usr/pppl/Modules/modulefiles ---------------------------------
	freetds/0.62.4(default)         lors/0.82(default)              totalview/6.4.0-0
	globus/2.4.3(default)           mdsplus/grid                    totalview/6.5.0-0
	globus/gt3.2.1                  mdsplus/gt3.2.1                 totalview/6.5.0-2
	hdf-view/2.0(default)           mdsplus/nongrid(default)        totalview/6.6.0-0
	ibp/1.3.1.2(default)            pathscale/2.0-197.2758(default) totalview/6.6.0-2(default)
	idl/6.0(default)                pdsh/2.3-1(default)             use.own
	iperf/1.7.0(default)            perl/5.8.5                      valgrind/2.2.0(default)
	java/1.4.2_05(default)          perl/5.8.6(default)             yorick/1.5.14(default)
	lff95/6.20c(default)            pgf/5.2-1(default)
	libxio/0.1(default)             pvfs2/1.0.1(default)

	----------------------------- /usr/pppl/Modules/compiler/lff95/6.20c ------------------------------
	fftw/2.1.5                 mpich2/1.0(default)        netcdfL/3.5.1(default)
	fftw/3.0.1(default)        nag/r3                     scalapack/1.7(default)
	hdf5/1.6.3-parallel        nag/r7(default)            superlu/3.0(default)
	hdf5/1.6.3-serial(default) ncarg/4.3.1(default)       yorick/1.5.14-parallel
	lapack-blas/3.0(default)   netcdf/3.6.0(default)
	mpich/1.2.6(default)       netcdf/3.6.0-beta6

One final thing you may notice with Modules is that loading a module may load other perquisite modules automatically. For example, MDSplus often requires use of IDL, so loading MDSplus will also load IDL if it's not loaded already.

	[me@portal ~]$ module load mdsplus
	[me@portal ~]$ module list
	Currently Loaded Modulefiles:
	  1) idl/6.0           2) mdsplus/nongrid

Login Scripts

In addition to Modules, there are other means of setting your environment on the Linux systems. Since the Linux systems running RedHat Enterprise Linux share home directories with the older UNIX machines at PPPL, the same login scripts will execute on both classes of systems. Chances are that your login scripts will require some tweaking to work correctly in both environments. Here is a simple test to run on portal to see if your login scripts could be generating a problem:

	[me@portal ~]$ $SHELL -l
	[me@portal ~]$ echo $PATH
	/usr/ad/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/pppl/bin
	[me@portal ~]$ echo $MANPATH
	/usr/share/man:/usr/X11R6/man:/usr/pbs/man:/usr/pppl/man
	[me@portal ~]$ exit
	logout
If you see any error message while doing this, such as '/etc/cshrc.default: No such file or directory', you need to update your login scripts. For details on this, please read our guide on updating your login scripts for the new systems.

Tips on Compiling Code

In addition to setting your PATH and MANPATH appropriately, Modules also sets several environment variables that are helpful in compiling code and constructing Makefiles. The following environment variables are set or pre-pended to as needed by each package:

Here are is an example of how to use the environment variables listed above to compile code. In this example, we've already loaded the modules that we need. The following command can be issued to compile a source file:

	[me@portal]$ $FC -I`echo $INCLUDE_PATH | sed -e 's/:/ -I/g'` -L`echo $LD_LIBRARY_PATH | sed -e 's/:/ -L/g'` -c -o myfile.o myfile.f
In the above command, $FC replaces having to call the compiler directly by name. This can help when trying code out under different compilers. INCLUDE_PATH and LD_LIBRARY_PATH are colon separated by default. However, most compilers expect additional paths to search for headers and libraries to be given individually, using either a -I or a -L option, respectively. So, sed is used to substitute the appropriate flags in path of the colons.

Submitting Jobs

For long running programs, and resource intensive applications, its a good idea to submit a job to PBS so that your program can have a dedicated set of machines to execute on. PBS is installed on all Linux nodes. For detailed information on how to use PBS, please take a look at the documentation. This documentation slightly out of date: first, you don't need to login to tern to submit jobs, you can submit jobs from portal. Secondly, in order get a new Linux node running RedHat Enterprise Linux, please submit your jobs to the falcon queue.

Printing

Printing is available on all Linux nodes using the CUPS printing system. CUPS provides a set of lp* commands, so users familar with BSD style LPD systems should have no problem using CUPS. There are several options that CUPS has added to commands like lpr, particularly some that allow you to control duplexing, paper size, and color. For information about the options available in CUPS, please take a look at the CUPS online manual.

One change from printing on PPPL's older UNIX systems is that there is only one queue per physical printer. On the older systems, sometimes there would be two or more queues available: one queue with duplexing enable, another without. CUPS allows the user to specify whether or not he wants duplexing on the command line, so having seperate queues to enable or disable features on the new systems is no longer needed. For example, to enable duplexing one can include -o sides=two-sided-short-edge as an argument to an lpr command.

The queue names in CUPS correspond exactly to those used by the desktop Windows and Mac systems at PPPL. For a list of available queue names, take a look at the /etc/printcap file on each machine - this file contains a list of available print queues. For detailed information about each queue, take a look at PPPL's print server. This web page has information such as where a printer is, what capabilites (color, duplexing, different paper sizes) are avialable, and for many of the printers, you can see the current status of the printer as well.

Finally, you can set a default printer by setting the PRINTER environment variable to the name of the printer you wish to use. For example, doing:

	[me@portal]$ export PRINTER=b143-xn2125
				
would make b143-xn2125 by default printer.

Getting Help

For assistance with the PPPL Linux systems, please send an email to unixadmin@pppl.gov. Be sure to be as detailed as possible in your problem description. For help getting your Mac or PC to connect to the Linux systems, you can also contact PPPL's Helpdesk via email at helpdesk@pppl.gov, or by calling extension 2275.