Running FACETS on multipole.txcorp.com

To connect to multipole:

ssh -X <username>@multipole.txcorp.com

Preamble

Running FACETS involves the following steps:

1. Correct setting of environment variables, make sure to have the correct Python module loaded.

2. Getting access to input data. These will typically consist of ".pre" files, likely g-eqdsk files for equilibrium data, and possibly other files required by individual components as well.

3. Preparing input data and running FACETS. These two steps are typically encoded into a single shell script, which takes care of running facets with the appropriate number of processes.

4. Analyzing and visualizing output data.

We will now guide you through each the above steps.

Accessing input data

svn co https://ice.txcorp.com/code/fctrain fctrain
cd fctrain

Configuring

First:

./config/cleanconf.sh

Then:

./configure \
 --with-serial-dir=/internal/facets \
 --with-parallel-dir=/internal/facets

This will produce the files:

components.sh  facetspath.sh  path.sh

Running core solo

cd runs/3-project-meeting/core-01
./core-01.sh

When the simulation completes it will produce files:

ls *.h5
core-01_0.h5  core-01_1.h5  core-01_2.h5  core-01_3.h5  core-01_4.h5  core-01_5.h5  fgCoreGrid_g118897.01555.h5

Plotting core results

To make the plots, we use python to read the HDF5 files. To make the plots we use the mkcoreplots.sh:

../code/mkcoreplots.sh -h
Usage
 -i Base name of run
 -s Start dump
 -n Number of dumps
 -h Help message

To plot the 5th dump at 1 ms do:

../code/mkcoreplots.sh -i core-01 -s 5 -n 1

This will produce a set of PNGS:

ls *.png
core-01_density_00005.png  core-01_temperature_H2p1_00005.png  core-01_temperature_electron_00005.png

You can use

eog core-01_density_00005.png

to plot the results.

Modifying the FACETS-core input data

We can modify for how long the simulation runs by editing the input file. Open the input file core-01.pre and change the variable TEND to a larger number, say 20e-3:

TEND = 20.e-3

Also, lets increase the number of processors to 8. For this edit the core-01.sh file and change to use 8 processors.

cmd="$MPILAUNCHER $MPIOPTS -n 8 $nodesarg $FACETSST -i $RUN_NAME.in -o ${RUN_NAME}_n8"

Note we have also changed the output file prefix so that the new results do not overwrite the old results. Save the input file and the shell script and launch the job again:

./core-01.sh

Running edge standalone

To run UEDGE drive through FACETS, do

cd runs/3-project-meeting/edge-01
./edge-01.sh

To monitor the job's progress:

qstat -a | grep $USER

Running core and edge concurrently

Now, we will run core-edge simulation concurrently. For this do

cd runs/3-project-meeting/core-edge-01
./core-edge-01.sh

Once the job completes we can plot the results using the mkplot

../code/mkplots.sh -h                       
Usage
 -i Base name of run
 -s Start dump
 -n Number of dumps
 -e Extra parameters to pas

To plot the final data (i.e. frame 5) at 1 ms hence do:

../code/mkplots.sh -s 5 -n 1 -i core-edge-01

This will produce a set of png images:

ls *.png
core-edge-01_coupled-lineout_density_00005.png           core-edge-01_coupled-lineout_temperature_electron_00005.png
core-edge-01_coupled-lineout_temperature_H2p1_00005.png

and data files:

ls *.dat 
core-edge-01_coupled-lineout_density_00005.dat               core-edge-01_edge-lineout_dif_use_00005.dat
core-edge-01_coupled-lineout_temperature_H2p1_00005.dat      core-edge-01_edge-lineout_kye_use_00005.dat
core-edge-01_coupled-lineout_temperature_electron_00005.dat  core-edge-01_edge-lineout_kyi_use_00005.dat

Either one can view the PNGs or use the data files for further analysis.