Downloading and building FACETS on franklin.nersc.gov
To connect to franklin:
ssh -X <username>@franklin.nersc.gov
Detailed build instructions for franklin are available at https://www.facetsproject.org/wiki/FranklinBuild
But running of FACETS as instructed below does not require you to have your own build.
A step by step guide to running FACETS on franklin.nersc.gov
Preamble
Running FACETS involves the following steps:
1. Correct setting of environment variables, make sure to have the correct Python module loaded.
2. Getting access to input data. These will typically consist of ".pre" files, likely g-eqdsk files for equilibrium data, and possibly other files required by individual components as well.
3. Preparing input data and running FACETS. These two steps are typically encoded into a single shell script, which takes care of running facets with the appropriate number of processes.
4. Analyzing and visualizing output data.
We will now guide you through each the above steps.
Accessing input data
cd $SCRATCH svn co https://ice.txcorp.com/code/fctrain fctrain cd fctrain
Set up your environment correctly. To find out which shell you are using:
echo $SHELL
If bash, type:
source rcfiles/franklin_pathscale.bashrc
if csh do:
source rcfiles/franklin_pathscale.cshrc
Configuring
First:
./config/cleanconf.sh
Then:
./configure \ --with-serial-dir=/project/projectdirs/facets/franklin/internal-path-3.2/facets \ --with-parallel-dir=/project/projectdirs/facets/franklin/internal-path-3.2/facets \ --with-fluxgrid=/project/projectdirs/facets/franklin/internal-path-3.2/fluxgrid/bin/fluxgrid \ --with-hdf5-dir=/project/projectdirs/facets/franklin/contrib-path-3.2/hdf5 \ --with-mpi-launcher=aprun
This will produce the files:
components.sh facetspath.sh path.sh
Running core solo
cd runs/3-project-meeting/core-01 qsub core-01.sh
To monitor the job's progress:
qstat -a | grep $USER
or, alternatively to monitor the run in real time:
tail -f core-01_0.log [Ctrl-C]
When the simulation completes it will produce files:
ls *.h5 core-01_0.h5 core-01_1.h5 core-01_2.h5 core-01_3.h5 core-01_4.h5 core-01_5.h5 fgCoreGrid_g118897.01555.h5
Plotting core results
To make the plots, we use python to read the HDF5 files. To make the plots we use the mkcoreplots.sh:
../code/mkcoreplots.sh -h Usage -i Base name of run -s Start dump -n Number of dumps -h Help message
To plot the 5th dump at 1 ms do:
../code/mkcoreplots.sh -i core-01 -s 5 -n 1
This will produce a set of PNGS:
ls *.png core-01_density_00005.png core-01_temperature_H2p1_00005.png core-01_temperature_electron_00005.png
You can use
xview core-01_density_00005.png
to plot the results.
Modifying the FACETS-core input data
We can modify for how long the simulation runs by editing the input file. Open the input file core-01.pre and change the variable TEND to a larger number, say 20e-3:
TEND = 20.e-3
Also, lets increase the number of processors to 8. For this edit the core-01.sh file and change to use 8 processors.
cmd="$MPILAUNCHER $MPIOPTS -n 8 $nodesarg $FACETSST -i $RUN_NAME.in -o ${RUN_NAME}_n8"
Note we have also changed the output file prefix so that the new results do not overwrite the old results. Save the input file and the shell script and launch the job again:
qsub core-01.sh
Running edge standalone
To run UEDGE drive through FACETS, do
cd $SCRATCH/fctrain/runs/3-project-meeting/edge-01 qsub edge-01.sh
To monitor the job's progress:
qstat -a | grep $USER
Running core and edge concurrently
Now, we will run core-edge simulation concurrently. For this do
cd $SCRATCH/fctrain/runs/3-project-meeting/core-edge-01 qsub core-edge-01.sh
To monitor the job's progress:
qstat -a | grep $USER
Once the job completes we can plot the results using the mkplot
../code/mkplots.sh -h Usage -i Base name of run -s Start dump -n Number of dumps -e Extra parameters to pas
To plot the final data (i.e. frame 5) at 1 ms hence do:
../code/mkplots.sh -s 5 -n 1 -i core-edge-01
This will produce a set of png images:
ls *.png core-edge-01_coupled-lineout_density_00005.png core-edge-01_coupled-lineout_temperature_electron_00005.png core-edge-01_coupled-lineout_temperature_H2p1_00005.png
and data files:
ls *.dat core-edge-01_coupled-lineout_density_00005.dat core-edge-01_edge-lineout_dif_use_00005.dat core-edge-01_coupled-lineout_temperature_H2p1_00005.dat core-edge-01_edge-lineout_kye_use_00005.dat core-edge-01_coupled-lineout_temperature_electron_00005.dat core-edge-01_edge-lineout_kyi_use_00005.dat
Either one can view the PNGs or use the data files for further analysis.
Running multiple gyro instances concurrently
Now, we will run gyro on multiple flux surfaces concurrently. For this do
cd $SCRATCH/fctrain/runs/3-project-meeting/gyro-01 qsub gyro-01.sh
To monitor the job's progress:
qstat -a | grep $USER
Now, once can look at the fluxes generated by gyro:
h5dump -d /facets/flux gyro-01_1.h5
This will write out the gyro fluxes written to the console.
