On Windows

Open a Command Prompt (run cmd.exe) and execute the following line:

C:\> %SCRIPT_DIR%\setupCmdEnv.bat

On Linux or Mac

In a bash shell, source the VSimComposer.sh script as follows:

$ source $SCRIPT_DIR/VSimComposer.sh

This is a bash shell script, which means you must be running the bash shell to execute the above command. If you are normally a csh/tsh user, you will need to start up a bash shell to execute the above command and subsequently execute VSimComposer.

Most distribution’s operating systems will allow you to add the above command to the .bashrc file in your home directory, which will prevent having to run it each time you log in. Any changes you make to your .bashrc do not take effect until the next time you log in, so after modifying your .bashrc file, you must execute the following command in your current shell, but will not need to do it in the future:

$ source ~/.bashrc

Running Vorpal with mpiexec

In order to run Vorpal in parallel via the command line, you must first add the <VORPAL_BIN_DIR> to your PATH, as noted in the Vorpal Command Line Options section.

To run Vorpal in parallel, execute the following command:

mpiexec -n <#> vorpal -i filename.pre

in which <#> is the number of processors, vorpal is the executable program for parallel computations, and filename.pre is the name of the input file (which must be in the current directory, or must be specified by a full path).

Following mpiexec, but before vorpal, you can specify a variety of mpiexec options. In particular, for the openmpi implementation of MPI (supplied with macOS), one may need to add the arguments, -x PYTHONPATH -x LD_LIBRARY_PATH to ensure that all processes are using the correct values for these environment variables. For the MPICH implementation of MPI (supplied with Linux) these arguments are not needed, as MPICH by default exports all environment variables to all processes. For more information about mpiexec, including the complete list of options, it can be run with mpiexec -h.

Following vorpal, you can specify a variety of Vorpal options. Some of the more common options are

vorpalser -i esPtclInCell.pre -o newesPtclInCell

vorpalser -i esPtclInCellSteps.pre -r 50

For a complete list of options, see VSim Reference: Running Vorpal from the Command Line Options.

If a parameter is both set within the input file and specified on the command line, the command line parameter value takes precedence. The command line override enables you to configure an input file with default values while exploring alternative parameter settings from the command line. From the command line, you can quickly change simulation run lengths, dimensionality, output timing, etc.

Vorpal automatically adjusts its decomposition to match the number of processors it is given, unless a manual decomposition is provided for the correct number of cores in the input .pre file.

In contrast to Vorpal for serial computation, which creates a single text output file, Vorpal for parallel computation creates multiple text output files. Each individual processor from the parallel run sends comments to a different output file. A parallel computation output file’s name includes a label that identifies the number of the processor that generated that file, for example:

• esPtclInCell_comms_0_1.txt
• esPtclInCell_comms_0_2.txt

in which the final _0 and _1 before the file name suffix indicate the number of the processor.

By default Vorpal writes one HDF5 file for each field or particle species even for a parallel run. However, one can modify this behavior, as noted at the above link. Having one file for each field or particle species for each processor can sometimes get around parallel I/O problems. When that is necessary, one can construct a single file for a field or particle species using the utilities, mergeH5Flds and mergeH5Ptcls, which come with Vorpal.

Running Vorpal with mpiexec Using a Hostfile

If the you need to run an MPI job but do not have access to a queuing system then a hostfile must be set up. If this is the case you must know the node names on the cluster that the job is to be run on. You must then create a text file with your text editor of choice, this is your hostfile, and place it in your home directory. The hostfile simply contains each node name repeated on a new line as many times as there are threads in that node. For example consider a two node cluster with four threads each, the hostfile will contain

node1
node1
node1
node1
node2
node2
node2
node2

To run a job one must then source VSimComposer shell script using the command:

source <VSIM_SCRIPT_DIR>/VSimComposer.sh

You are now ready to run in MPI using the mpiexec command with the above hostfile (signified as <hostfile> below. For MPICH (which is provided for Linux), the command is

mpiexec -f <hostfile> -n <#> <other mpiexec options> vorpal \
  -i simulationname.pre <other vorpal options>

The equivalent command for openmpi (which is provided for macOS) is

mpiexec --hostfile <hostfile> -n <#> <other mpiexec options> vorpal \
  -i simulationname.pre <other vorpal options>

The number of nodes, <#>, must be consistent with the computational resources and the hostfile.