NOTE: VASP is proprietary code. To use VASP on the NNIN system you must demonstrate that you possess a license for VASP.
These are directions for building and running VASP with Intel v9.1 compilers and MPICH on the new NNIN cluster, chassis 1,3 & 4.
NOTE: VASP under condor requires that the executable be located in your executing directory. Thus you should copy /group/nnin/VASP/bin/vasp.parallel to your home and keep it in the directory where you expect to do I/O. It is possible that you can alias the executable but that has not been tested yet. Please inform NNIN/C (we will have a VASP wiki up soon) if you learn anyting about using aliases.
NOTE: The current VASP under condor installation requires that your login shell be bash. It is therefore not sufficient to just change to bash. Please follow the instructions on changing login shell to VASP.
NOTE: VASP will not run in this way under direct mpirun. Currently we are modifying the system and we will include the ability to run jobs (for testing purposes) directly with mpirun in the future.
First, check to see that your bash configuration file, .bashrc, contains the following statement as the last statement in the file:
module load null compilers/intel/9.1 libraries/acml/4.0/intel mpi/mpich/1.2.7p1/intel9
If it is not there it is simplest just to edit the file and add this line (at the end). You can also look at the write-up for modules and use initadd.
Now, you should check to see that the modules are loaded (this will not be the case if you have modified the .bashrc file without logging off and loggin on again). To do this, do a
>module list
and you should see the list of the three modules that are listed in your /bashrc file (not including, of course, “null”). If they are not there for some reason, do the following (cf. modules):
> module load compilers/intel/9.1
> module load libraries/acml/4.0/intel
> module load mpi/mpich/1.2.7p1/intel9
(N.B. in “1.2.7p1” above: last character is a one not an el. Check these using module avail to make sure the names are correct) .
To make this the default shell environment, use the "initadd" instead of "load" subcommand in the above (see modules write-up for details).
RUN
If you have a license to run VASP, you can use the executable in /group/nnin/VASP/bin
entitled: vasp.parallel-mpich. You should set up your environment as indicated above, being sure that you have the mpich module and not the openmpi module. You may have to unload the openmpi module if you are using the .cshrc file available in /group/nnin/VASP/ (module unload mpi/openmpi/1.2.4/intel9). See module description for details.
The executable is stored in /group/nnin/VASP/bin/vasp.parallel and is accessible only to licensed VASP users. The example input files and condor submit script, “vasp.submit,” are stored in /group/nnin/VASP/condor/zone. Copy all of the contents of the zone directory to an otherwise empty directory under your home directory. Copy the executable in /group/nnin/VASP/bin/vasp.parallel to the same working directory. Modify the path in the script vasp.submit so that the executable is the one in your home directory. This is crucial ! Execute vasp with condor:
>condor_submit vasp.submit.
You should have output files in your directory in about two minutes. Change the input files to suit your case. You can also change the vasp.submit script to, for example, run on more processors (current default is 8) but the more you choose the longer it will take to run, so an optimum based on cluster usage exists.
Details: the vasp.submit script does not call the executable directly but rather calls a vasp wrapper script called vasp-mpirun located in /opt/condor/bin which you can look at if you want to. The script vasp.submit passes the name of the executable to the wrapper script along with other parameters such as the number of processors desired.
NOTE: Vasp currently runs, using condor, under mpich but not openmpi.
NOTE: Hopefully in the future the need to keep the executable in the working directory will be circumvented.
BUILD
If you wish to build your own version of VASP (modifying the source code, for example), here are the directions.
Grab the makefile: /group/nnin/VASP/condor/build/Makefile.mpi for building VASP with MPI support (note that this will probably want to be modified to enable better optimizations and add better libraries in the future). This makefile will allow you to build a parallel version of VASP under intel compilers version 9.1. IMPORTANT: you will first have to prepare your environment (i.e. load modules) as indicated above in the ”RUN” section.
Build the vasp.4.lib library:
> cd VASP/src/vasp.4.lib
> rm *.o *.a
> make
Then VASP itself
> cd ../vasp.4.6
> make clean
> make -f Makefile.mpi
This should generate the vasp binary. I'd rename it to something like "vasp.parallel" to reduce confusion.
Next, note that VASP will not run under direct mpirun submission. Rather, you must use condor.
To run under condor, check that the submit file has "getenv = true" (in the /group/nnin/VASP/zone submit file it is already set). If so, then it'll pick up the set environment.
Use "executable = /your path/vasp.parallel" which is the path to where both the executable, this script and the input decks are (and where the output decks will go).
The version of mpirun used will be the version in your present environment. Thus modules controls how condor runs the job, because of the "getenv = true" setting in the submit file.
|
