This document is geared towards helping users with some Linux or UNIX experience learn the details of the Linux systems at PPPL specific to our site. If you're new to the Linux world, please take a look at http://www.tldp.org/LDP/intro-linux/html/ first for a tutorial on the basics, before continuing on with this guide. For the more seasoned Linux user, please be sure to read through this guide, as it contains many tips that will help you work efficiently on PPPL's systems.
The main entry point to the PPPL Linux environment is a host called portal.pppl.gov. Portal is not
an actual host, but rather an alias to a pool of nodes available for general login sessions. The name portal will
automatically adjust itself to point to the best (least loaded) system in the interactive portal pool. The portal nodes
all have publicly accessible IP addresses; you will be to access them directly from just about anywhere assuming that you've
authenticated with PPPL's firewall first.
So you use portal as your entry point to PPPL's Linux systems.
There are many other Linux hosts contained in clusters, and not for direct logins. These hosts are reserved by submitting a job to a job scheduler. The job scheduler will determine what hosts you need for your job, then, once the hosts are available, start the job. See hardware for information about the systems available for running jobs.
All nodes at PPPL run a 64 bit version of Linux based on RedHat Enterprise Linux V 5, including all of the general use nodes in portal.
There is a small group of nodes running the previous version of Linux in a group called portalr4, which is a pool of
general use machines just like portal. To access these previous version systems, ssh to
portalr4.pppl.gov.
All Linux hosts running RedHat Enterprise Linux can be accessed remotely via SSH. SSH is the only
mechanism for accessing these nodes - other protocols such as Telnet, RSH, REXEC, RCP, FTP, X, XDM, and XDMCP are not available due to security
concerns. You will however, be able to run X applications using X forwarding included with SSH. All users are allowed to login to the
portal nodes via SSH at any time. Even though the other Linux hosts have SSH enabled, users are not allowed to login directly,
unless that user reserved the node via the job scheduler. Once a node is reserved with the job scheduler, that user will
automatically be granted access to directly login to a computational node for the duration of their job.
Almost all the file systems on the PPPL Linux systems are served from a central server. Here is list of what file systems are available, what they hold, and who can access them:
/tmp is local disk mounted on every node. This area is used by many programs for
short term storage of working data. /tmp is writable
by everyone, but is only a few gigabytes in size, typically double
the size of memory. Users are strongly discouraged from intentionally
storing data there, as this area is wiped clean every time the system
is booted, and data in /tmp is not backed up.
/local is another file system, writeable by everyone
like /tmp. Unlike /tmp, /local
is considerably larger (the size of the disk minus /tmp and swap), and
is not wiped out when the system reboots. Users are encouraged to use
/local as place to store their work while they are working.
Since /local is a local disk (and not a network disk),
file access times will be quite good. /local is writeable
by any user, but only the user that owns files in /local
can delete them. Beware that /local is not a long term
storage solution. Data in this area is not backed up, and a script will
run periodically to remove file to maintain free space. Also, if problems
arise, such as a corrupt file system or failed disk, the data in
/local may be lost.
/l is a network file system, which allows a user to access data on another system's /local disk.
Under /l, you will find a directory corresponding to every Linux hosts' name. Within the host name directories,
you will have access via NFS to that host's /local disk. For example, if I wanted to access the /local
disk on the host dawson001 from a portal node, I would do:
[me@portal ~]$ cd /l/dawson001 [me@portal /l/dawson001]$ ls my.files*
/l is accessible from all Linux systems. Like /local, all the hostname directories within /l
are writeable by the file owners.
/p contains project space, which is large, long term network storage. This area is stored on central network servers and
accessible from all systems. Generally speaking, you will get slightly
better performance for your job by working on files in /local rather than
in /p, and then, once the job completes, data is the moved from
/local to your project space so that the information
can be shared with others.
Project space is backed up. Requests to get started using project space should be directed to the
system administrators.
/u contains the user home directories. This filesystem is
stored on central network servers and accessible from all systems.
Paths to home directories are of the form /u/username.
Home directories are backed up, and are a good place to store
your personal files, but not project data. Home directories do have
much smaller quotas than project disks or /local. If you
need a lot of space to store data, please ask for a project disk.
/usr/pppl is a central filesystem served from network servers
that holds many of the applications (compilers, mathematics packages, etc.)
we have at PPPL. For information on
how to take advantage of /usr/pppl, please see the section
on finding programs.
On the PPPL Linux systems, programs can be divided into two categories: those
provided as part of Linux, and those installed by PPPL. Programs included with the
system, such as awk, grep, sed, vi,
bash, gcc, autoconf, and xterm,
are installed in the standard location, usually /bin
or /usr/bin. Likewise, the basic system libraries and headers are
found in places like /lib and /usr/include.
Programs installed by PPPL administrators, such as code compilers, IDL, MPICH, and
MDSplus clients are found in /usr/pppl. These programs are installed in
directories beneath /usr/pppl, dependent on the package name and the
version. For example, version 6.20c of Lahey-Fujitsu Fortran can be found in
/usr/pppl/lff95/6.20c. This allows PPPL to maintain different versions
of packages at the same time. The version specific installation path paradigm is
taken even further in the case of libraries that depend on a specific compiler.
For example, if one wanted to use NetCDF, one would find the NetCDF installation directory within the installation folder of the compiler used to
build NetCDF. NetCDF 3.6.0 for Lahey-Fujitsu Fortran can be found in /usr/pppl/lff95/6.20c/netcdf-3.6.0 and NetCDF for the PathScale
v2.0-197.2758 compiler is found in /usr/pppl/pathscale/2.0-197.2758/netcdf-3.6.0. This scheme allows both administrators and users
to avoid mixing different versions of a package - i.e. it becomes very difficult to get the man packages for one version of IDL and the binaries
from a different version.
In order to help users navigate around /usr/pppl, we've setup a package called Modules.
As was mentioned in the section above, /usr/pppl has a unique path for every piece of software installed by PPPL. In order to
make it easy to manage all the different paths that software can be found in, PPPL has setup a package used at many supercomputing sites called
Modules. Modules does what the setup scripts (such as those included with IDL or
Lahey-Fujitsu Fortran) did - it sets environment variables, updates your PATH, and sets aliases in your shell. Modules also has an 'unload'
capability; after you're done using a package, you can return your environment to its previous state. This feature is important if you
wish to switch from one version of a package to another.
Here is a tutorial on how to use Modules: Then first command to learn is 'module avail' - this will show you a list of packages that
are currently available for you to use:
[me@portal ~]$ module avail ----------------------------------- /usr/pppl/Modules/modulefiles --------------------------------- freetds/0.62.4(default) lors/0.82(default) totalview/6.4.0-0 globus/2.4.3(default) mdsplus/grid totalview/6.5.0-0 globus/gt3.2.1 mdsplus/gt3.2.1 totalview/6.5.0-2 hdf-view/2.0(default) mdsplus/nongrid(default) totalview/6.6.0-0 ibp/1.3.1.2(default) pathscale/2.0-197.2758(default) totalview/6.6.0-2(default) idl/6.0(default) pdsh/2.3-1(default) use.own iperf/1.7.0(default) perl/5.8.5 valgrind/2.2.0(default) java/1.4.2_05(default) perl/5.8.6(default) yorick/1.5.14(default) lff95/6.20c(default) pgf/5.2-1(default) libxio/0.1(default) pvfs2/1.0.1(default)The next command to learn is '
module list'. This will show you
what modules you currently have loaded.
[me@portal ~]$ module list No Modulefiles Currently Loaded.As you can see, no modules are currently loaded in my environment. To load module, such as perl, I can simply issue the following command:
[me@portal ~]$ module load perl [me@portal ~]$ module list Currently Loaded Modulefiles: 1) perl/5.8.6 [me@portal ~]$ which perl /usr/pppl/perl/5.8.6/bin/perlThe default version of perl, 5.8.6, was loaded into my environment by simply loading 'perl'. If I specifically, wanted to have version 5.8.5, instead of the default version 5.8.6 perl, I could have issued '
module load perl/5.8.5'. Modules also have conflicts; for example
I can only have one version of perl loaded at a time. If I already have loaded the default (5.8.6) version of perl, and I then try to load
version 5.8.5 of perl, I will get the following error:
[me@portal ~]$ module list Currently Loaded Modulefiles: 1) perl/5.8.6 [me@portal ~]$ module load perl/5.8.5 perl/5.8.5(8):ERROR:150: Module 'perl/5.8.5' conflicts with the currently loaded module(s) 'perl/5.8.6' perl/5.8.5(8):ERROR:102: Tcl command execution failed: conflict perlIn order to switch to using another version of perl, I can use '
module switch':
[me@portal ~]$ module list Currently Loaded Modulefiles: 1) perl/5.8.6 [me@portal ~]$ module switch perl perl/5.8.5 me@portal ~]$ module list Currently Loaded Modulefiles: 1) perl/5.8.5 [me@portal ~]$ which perl /usr/pppl/perl/5.8.5/bin/perlTo unload a module, I can use '
module unload':
[me@portal ~]$ module unload perlFinally, if I wish to remove all modules, I issue '
module purge':
[me@portal ~]$ module purge
A special case with Modules is the way that compiler dependant libraries are handled. Things like MPICH and NetCDF are built with a certain compiler, and require that compiler to function properly. Because of this relationship, the compiler dependant functions do not show up on the initial list of available modules. However, once you load the module for a compiler, an additional subset of modules specifically for that compiler will appear:
[me@portal ~]$ module avail ----------------------------------- /usr/pppl/Modules/modulefiles --------------------------------- freetds/0.62.4(default) lors/0.82(default) totalview/6.4.0-0 globus/2.4.3(default) mdsplus/grid totalview/6.5.0-0 globus/gt3.2.1 mdsplus/gt3.2.1 totalview/6.5.0-2 hdf-view/2.0(default) mdsplus/nongrid(default) totalview/6.6.0-0 ibp/1.3.1.2(default) pathscale/2.0-197.2758(default) totalview/6.6.0-2(default) idl/6.0(default) pdsh/2.3-1(default) use.own iperf/1.7.0(default) perl/5.8.5 valgrind/2.2.0(default) java/1.4.2_05(default) perl/5.8.6(default) yorick/1.5.14(default) lff95/6.20c(default) pgf/5.2-1(default) libxio/0.1(default) pvfs2/1.0.1(default) [me@portal ~]$ module load lff95 [me@portal ~]$ module avail ----------------------------------- /usr/pppl/Modules/modulefiles --------------------------------- freetds/0.62.4(default) lors/0.82(default) totalview/6.4.0-0 globus/2.4.3(default) mdsplus/grid totalview/6.5.0-0 globus/gt3.2.1 mdsplus/gt3.2.1 totalview/6.5.0-2 hdf-view/2.0(default) mdsplus/nongrid(default) totalview/6.6.0-0 ibp/1.3.1.2(default) pathscale/2.0-197.2758(default) totalview/6.6.0-2(default) idl/6.0(default) pdsh/2.3-1(default) use.own iperf/1.7.0(default) perl/5.8.5 valgrind/2.2.0(default) java/1.4.2_05(default) perl/5.8.6(default) yorick/1.5.14(default) lff95/6.20c(default) pgf/5.2-1(default) libxio/0.1(default) pvfs2/1.0.1(default) ----------------------------- /usr/pppl/Modules/compiler/lff95/6.20c ------------------------------ fftw/2.1.5 mpich2/1.0(default) netcdfL/3.5.1(default) fftw/3.0.1(default) nag/r3 scalapack/1.7(default) hdf5/1.6.3-parallel nag/r7(default) superlu/3.0(default) hdf5/1.6.3-serial(default) ncarg/4.3.1(default) yorick/1.5.14-parallel lapack-blas/3.0(default) netcdf/3.6.0(default) mpich/1.2.6(default) netcdf/3.6.0-beta6
One final thing you may notice with Modules is that loading a module may load other
perquisite modules automatically. For example,
MDSplus often requires use of IDL, so loading MDSplus will also load IDL if
it's not loaded already.
[me@portal ~]$ module load mdsplus [me@portal ~]$ module list Currently Loaded Modulefiles: 1) idl/6.0 2) mdsplus/nongrid
In addition to Modules, there are other means of setting your environment on the Linux systems. Since the
Linux systems running RedHat Enterprise Linux share home directories with the older UNIX machines at PPPL, the same login scripts
will execute on both classes of systems. Chances are that your login scripts will require some tweaking to work correctly in both
environments. Here is a simple test to run on portal to see if your login scripts could be generating a problem:
[me@portal ~]$ $SHELL -l [me@portal ~]$ echo $PATH /usr/ad/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/pppl/bin [me@portal ~]$ echo $MANPATH /usr/share/man:/usr/X11R6/man:/usr/pbs/man:/usr/pppl/man [me@portal ~]$ exit logoutIf you see any error message while doing this, such as '
/etc/cshrc.default: No such file or directory', you need to update
your login scripts. For details on this, please read our guide on updating your login scripts for the new
systems.
In addition to setting your PATH and MANPATH appropriately, Modules also sets several environment variables that are helpful in compiling code and constructing Makefiles. The following environment variables are set or pre-pended to as needed by each package:
Here are is an example of how to use the environment variables listed above to compile code. In this example, we've already loaded the modules that we need. The following command can be issued to compile a source file:
[me@portal]$ $FC -I`echo $INCLUDE_PATH | sed -e 's/:/ -I/g'` -L`echo $LD_LIBRARY_PATH | sed -e 's/:/ -L/g'` -c -o myfile.o myfile.fIn the above command,
$FC replaces having to call the compiler directly by name. This can help when trying code out
under different compilers. INCLUDE_PATH and LD_LIBRARY_PATH are colon
separated by default. However,
most compilers expect additional paths to search for headers and libraries to be given
individually, using either a -I
or a -L option, respectively. So, sed is used to substitute the appropriate flags in path of the colons.
Printing is available on all Linux nodes using the CUPS printing system. CUPS provides a set of
lp* commands, so users familar with BSD style LPD systems should have no problem using CUPS. There are several options
that CUPS has added to commands like lpr, particularly some that allow you to control duplexing, paper size, and color.
For information about the options available in CUPS, please take a look at the
CUPS online manual.
One change from printing on PPPL's older UNIX systems is that there is only one queue per physical printer. On the older systems,
sometimes there would be two or more queues available: one queue with duplexing enable, another without. CUPS allows the user to specify
whether or not he wants duplexing on the command line, so having seperate queues to enable or disable features on the new systems is
no longer needed. For example, to enable duplexing one can include -o sides=two-sided-short-edge as an argument to an
lpr command.
The queue names in CUPS correspond exactly to those used by the desktop Windows and Mac systems at PPPL. For a list of available
queue names, take a look at the /etc/printcap file on each machine - this file contains a list of available print queues.
For detailed information about each queue, take a look at PPPL's print server. This web page
has information such as where a printer is, what capabilites (color, duplexing, different paper sizes) are avialable, and for many of the
printers, you can see the current status of the printer as well.
Finally, you can set a default printer by setting the PRINTER environment variable to the name of the printer you wish to use.
For example, doing:
[me@portal]$ export PRINTER=b143-xn2125would make
b143-xn2125 by default printer.
For assistance with the PPPL Linux, Mac and PC systems, please use our online ticketing system service now. On the side tab under incident select Create New. As 'Assignment group' for Linux distributions please select SN_Linux and for Mac and PC select SN_Helpdesk. Be sure to be as detailed as possible in your problem description and if possible supply error messages and commands that were used to produce the error.