# Solving Problems on Unstructured Meshes in USim¶

In Using USim to solve the Euler Equations we discussed the basic methods used by USim to solve the Euler equations. Next, in Using USim to solve the Magnetohydrodynamic Equations, we extended these ideas to solve the MHD equations in one-dimension and showed how USimBase simulations for both the Euler and MHD equations follow the same basic pattern. Then, in Solving Multi-Dimensional Problems in USim, we built on these concepts to demonstrate how to use USim to solve the Euler and MHD equations in multi-dimensions, how to utilize more advanced boundary conditions and how to apply external fources (such as gravity) to the equations. Following on from this in Solving Problems on Advanced Structured Meshes in USim, we demonstrated how USim can solve problems in axisymmetric curvilinear coordinates and how to use two- and three-dimensional body fitted meshes to solve problems around simple geometries. In this tutorial, we introduce unstructured meshes in USim, show how to create boundary conditions for simple geometries, apply custom boundary conditions based on user specified parameters, and compute flow diagnostics.

The Flow over a Forward-Facing Step (forwardFacingStep.pre) example in USimBase illustrates how to solve problems on unstructured meshes.

Note

In USim Version 3.0 unstructured meshes should be quadrilateral or hexahedral. If unstructured grids are used in parallel then the user needs to define the partitioning of the grid prior to import; USim does not partition unstructured meshes itself.

## Creating Unstructured Meshes in CUBIT for USim¶

USim currently accepts Exodus II files created in a number of programs including CUBIT and Trelis. Exodus II files generally have the extension .exo, however we frequently use the genesis format (.g) which contains only the mesh data of the Exodus II file.

USim 3.0 currently accepts only quad or hex elements.

• First, create a mesh in CUBIT following the CUBIT tutorials.
• After the mesh has been created, sidesets (used for boundaries) can also be created and then used in defining the locations of boundary conditions.
• In CUBIT, select ‘Materials and Bcs’ then select ‘Entities - Blocks’. From the drop down menu select ‘Add’ and set the ‘Block ID’ to 1. If the mesh is 2d pick ‘Surface’ and hit ‘Apply’. If the mesh is 3D, instead select ‘Volume’ and hit ‘Apply’.
• Next, in ‘Entities - Blocks’ switch the drop down menu to ‘Element Type’. If the mesh is 2D select ‘Surfaces’ and then ‘Quad4’, if the mesh is 3D select ‘Volumes’ and ‘Hex8’, hit ‘Apply’.
• Next switch from ‘Entity - Blocks’ to ‘Entity - Sidesets’. In 2D select ‘Curve’ then click on the mesh or the boundary you wish to choose for your sideset. In 3D select ‘Surface’ then select the boundary you are interested in. You will now have a boundary defined as sideset 1 (for example).
• Once the grid is read into USim an entity is defined over the side with the name sideSetHalosId1. This entity can then be used USim boundary conditions.
• Add any other side sets you wish to use as boundaries at this point. The mesh is now ready to export.
• In the main CUBIT window select ‘File->Export’. Choose the directory to export the file and from the dropdown menu select ‘Genesis’ to export a Genesis file. Type in the file name and select ‘Save’. You should now have a grid that can be read into USim.

## Creating Unstructured Meshes in Trelis for USim¶

USim currently accepts Exodus II files created in a number of programs including CUBIT and Trelis. Exodus II files generally have the extension .exo, however we frequently use the genesis format (.g) which contains only the mesh data of the Exodus II file.

USim 3.0 currently accepts only quad or hex elements.

• First, create a mesh in Trelis using tutorials and documentation.
• After the mesh has been created, sidesets (used for boundaries) can also be created and then used in defining the locations of boundary conditions.
• In Trelis, select the ‘Analysis Groups and Materials’ mode, then select ‘Entity-Sidesets’.
• Set the ‘Action’ to ‘Create sideset’ and set the ‘Sideset ID’ to 1
• Select ‘Curve’ if the mesh is 2D and ‘Surface’ if the mesh is 3D and type in the curve/surface ID or select it with your mouse and hit ‘Apply’
• Once the grid is read into USim an entity is defined over the side with the name sideSetHalosId1. This entity can then be used USim boundary conditions.
• Add any other side sets you wish to use as boundaries at this point. The mesh is now ready to export.
• In the main Trelis window select ‘File->Export’. Choose the directory to export the file and from the dropdown menu select ‘Exodus’ to export an Exodus file. Type in the file name and select ‘Save’. You should now have a grid that can be read into USim.

## Creating Unstructured Meshes in Gmsh for USim¶

USim also accepts Gmsh meshes with extension .msh.

• First, create a geometry in Gmsh following any Gmsh tutorials.
• Gmsh will mesh using triangles/tets unless instructed otherwise. Open the ‘Tools -> Options’ menu, and under the ‘General’ tab for ‘Mesh’, check the box “Recombine all triangular meshes”. Also set the subdivision algorithm to “All Quads” or “All Hexes” for 2D or 3D meshes.
• Save the mesh. You should now have a grid that can be read into USim.

Gmsh does not have a facility for setting sideSets used for USim boundary conditions. Instead, an entity can be created in USim by masking off ghost cells to isolate those you would like to create a boundary condition on.

An example of masking off a region of the ghost cells is as follows:

<Updater generateOpenBc>
kind = entityGenerator2d
onGrid = domain
newEntityName = openBc
onEntity = ghost
kind = exprFunc
exprs = ["if(x>0 && y>0.03,1.0,-1.0)"]
</Function>
</Updater>


This block creates a new entity of name “openBC” which can be called in an Updater and used for setting a boundary condition.

## Running Unstructured Meshes in Parallel¶

### CUBIT/Trelis¶

If you wish to run your simulation in parallel, the mesh will need to be decomposed prior to use in USim. CUBIT and Trelis do not have a facility to do this, so an external program must be used. We recommend using SEACAS, which is a suite of applications for supporting finite element analysis software using Exodus file format. SEACAS Specifically, nem_slice and nem_spread.

In the Linux distribution of USim 3.0, an Exodus II mesh file can be decomposed using the provided partitioner script, decomp.

For example, if you plan to use 8 processors, from the command line run:

<ULIXES_BIN_DIR>/decomp -p 8 --root ./ meshfile.g


or

<ULIXES_BIN_DIR>/decomp -p 8 --root ./ meshfile.exo


<ULIXES_BIN_DIR>/decomp -h


A number of files will be output corresponding to the number of processors you plan to run on. In the case above, 8 files, namely:

meshfile.g.8.0 meshfile.g.8.1 meshfile.g.8.2 meshfile.g.8.3 meshfile.g.8.4 meshfile.g.8.5 meshfile.g.8.6 meshfile.g.8.7

USim will automatically detect these files upon running with 8 cores. In your <Creator> block, you simply need to set the file to ‘meshfile.g’:

<Creator ctor>
kind = exodus
ndim = 3
file = meshfile.g
</Creator>


### Gmsh¶

Gmsh, on the other hand, can partition your grid prior to saving. In Gmsh, select ‘Mesh->Partition’ and set the ‘Number of Partitions’ to however many cores you would like to run on. Then hit ‘Partition’. The colors in your mesh should change to show the decomposition of the mesh onto your chosen number of partitions.

Saving this mesh will result in 1 file being written. We recommend you save the file with the number of partitions in the file name.:

meshfile16.msh


USim will NOT automatically detect which file to use when running in parallel. In your <Creator> block, be sure to set the correct meshfile.:

<Creator ctor>
kind = gmsh
ndim = 3
file = meshfile16.msh
</Creator>


## Initializing a Simulation¶

One of the great strengths of USim is that the underlying algorithms are able to work on both structured and unstructured meshes. This means that the worflow to setup a simulation on a unstructured mesh is very similar to that seen previously:

# Are we solving the MHD equations?
$MHD = False # Import macros to setup simulation$ import fluidsBase.mac
$if MHD$ import idealmhd.mac
$else$ import euler.mac
$endif # Specify parameters for the specific physics problem$ PARAM_1 = <value>
$PARAM_2 = <value>$ PARAM_N = <value>

# Initialize a USim simulation
$if MHD initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,MU0,WRITE_RESTART,DEBUG)$ else
initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,WRITE_RESTART,DEBUG)
$endif  For the Forward Facing Step example with the Euler equations, the parameters that we specify for the physics problem are as follows: # grid for simulation$ GRIDFILE = "forwardFacingStep"
# format of grid
$GRIDTYPE = "gmsh" # adiabatic index$ GAS_GAMMA = 1.4
# Magnetic field strength
$BETA = 1.0e3 # end time for simulation$ TEND = 4.0
# number of frames
$NUMDUMPS = 20 # Riemann solver$ DIFFUSIVE = True
# Order in time
$TIME_ORDER = "second" # Write data for restarting the simulation$ WRITE_RESTART = False
# Output info for debugging purposes
$DEBUG = False # Dimensionality$ NDIM = 2
# Permeability of free space
$MU0 = 1.0 # CFL condition$ CFL = 0.4


The use of an unstructured mesh in USim is specified through the use of one of the two grids:

addExodusGrid(GRIDFILE)


or:

addGmshGrid(GRIDFILE)


The choice of which block to use corresponds to the format of the mesh; either GMSH or ExodusII format. The GRIDFILE is the name of the file containing the mesh without the file extension.

Our simulation input file now looks like:

# Are we solving the MHD equations?
$MHD = False # Import macros to setup simulation$ import fluidsBase.mac
$if MHD$ import idealmhd.mac
$else$ import euler.mac
$endif # Specify parameters for the specific physics problem # grid for simulation$ GRIDFILE = "forwardFacingStep"
# format of grid
$GRIDTYPE = "gmsh" # adiabatic index$ GAS_GAMMA = 1.4
# Magnetic field strength
$BETA = 1.0e3 # end time for simulation$ TEND = 4.0
# number of frames
$NUMDUMPS = 20 # Riemann solver$ DIFFUSIVE = True
# Order in time
$TIME_ORDER = "second" # Write data for restarting the simulation$ WRITE_RESTART = False
# Output info for debugging purposes
$DEBUG = False # Dimensionality$ NDIM = 2
# Permeability of free space
$MU0 = 1.0 # CFL condition$ CFL = 0.4

# Initialize a USim simulation
$if MHD initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,MU0,WRITE_RESTART,DEBUG)$ else
initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,WRITE_RESTART,DEBUG)
$endif$ if isEqualString(GRIDTYPE,ExodusII)
$else addGmshGrid(GRIDFILE)$ endif


## Creating a Fluid Simulation¶

As in the Initializing a Simulation step, the procedure for setting up a fluid simulation on an unstructured mesh is identical to that on a structured mesh. For the Flow Over a Forward Facing Step example, this proceeds as follows. First, we create the variables needed to simulate the fluid:

createFluidSimulation()


We then proceed through the three-step process to specify the distribution of the fluid on the grid. Step 1: Add Variables:

addVariable(gasGamma,GAS_GAMMA)


addPreExpression(rho = gasGamma)


Step 3: Add Expressions for density, momentum and total energy:

addExpression(rho)


Note that Step 3 is identical to that used in Using USim to solve the Euler Equations. Our simulation input file now looks like:

# Are we solving the MHD equations?
$MHD = False # Import macros to setup simulation$ import fluidsBase.mac
$if MHD$ import idealmhd.mac
$else$ import euler.mac
$endif # Specify parameters for the specific physics problem # grid for simulation$ GRIDFILE = "forwardFacingStep"
# format of grid
$GRIDTYPE = "gmsh" # adiabatic index$ GAS_GAMMA = 1.4
# Magnetic field strength
$BETA = 1.0e3 # end time for simulation$ TEND = 4.0
# number of frames
$NUMDUMPS = 20 # Riemann solver$ DIFFUSIVE = True
# Order in time
$TIME_ORDER = "second" # Write data for restarting the simulation$ WRITE_RESTART = False
# Output info for debugging purposes
$DEBUG = False # Dimensionality$ NDIM = 2
# Permeability of free space
$MU0 = 1.0 # CFL condition$ CFL = 0.4

# Initialize a USim simulation
$if MHD initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,MU0,WRITE_RESTART,DEBUG)$ else
initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,WRITE_RESTART,DEBUG)
$endif$ if isEqualString(GRIDTYPE,ExodusII)
$else addGmshGrid(GRIDFILE)$ endif

# Create data structures needed for the simulation
createFluidSimulation()

# Step 3: Add expressions specifying initial condition on density,
# momentum, total energy


## Evolving the Fluid¶

For multi-dimensional physics problems on structured meshes, our general pattern for evolving the fluid took the form:

# Add the spatial discretization of the fluxes
finiteVolumeScheme(DIFFUSIVE)

# Add (optional) physics to the finite volume scheme
< physics macros >

# Boundary conditions
boundaryCondition(<boundaryCondition,entity>)

# Time integration


## Boundary Conditions on Unstructured Meshes¶

The complexity of performing calculations on an unstructured mesh in USim is associated with application of boundary conditions. USim’s approach to applying boundary conditions on an unstructured mesh is a two-step process:

1. The user defines the regions of the mesh (entity) that will be used to apply boundary conditions.
2. The user specifies the boundary conditions to apply on each region of the mesh.

For the specific case of the Flow Over a Forward Facing Step example, there are three boundaries that we need to define, which are delimited by position in the streamwise direction, $$x$$:

1. Inflow boundary: defined for $$x < 0.0$$
2. Wall boundary: defined for $$0.0 < x < 3.0$$
3. Outflow boundary: defined for $$x > 3.0$$

We can generate boundary entities that exist on the exterior of the mesh using a combination of the createNewEntityFromMask and addEntityMaskExpression macros:

createNewEntityFromMask(<entityName>)


Using this combination of macros will result in an entity with name <entityName> being generated when <logicalExpression> expression evaluates to 1. For the three entities needed for the Flow Over a Forward Facing Step, these operations take the form:

# Inflow for x < 0.0

# Wall for 0.0 < x < 3.0

# Outflow for x > 3.0


Now that we have created the inflowEntity, wallEntity and outflowEntity, we can specify boundary conditions on them. For the wallEntity and the outflowEntity, the boundary conditions are familiar from our previous tutorials:

boundaryCondition(wall,wallEntity)
boundaryCondition(copy,outflowEntity)


For the inflowEntity, we specify a new type of boundary condition userSpecified to determine the inflow properties:

boundaryCondition(userSpecified,inflowEntity)


The userSpecified boundary condition allows the user to specify the properties of the flow on the boundary entity (here inflowEntity) using what should now be a familiar three-step process:

1. Use addBoundaryConditionVariable to add variables that are independent of grid position.
2. Use addBoundaryConditionPreExpression to add quantities that are functions of grid position, variables and any previously defined PreExpression in this block. Evaluated before expressions and the result is not accessible outside of this block. Any number of PreExpressions can be added.
3. Use addBoundaryConditionExpression to define each boundarycondition for the fluid. There is one expression for density, each component of momentum and the total energy. The order of the exprssions correspond to the order in the state vector and there can only be one expression per entry in the state vector.

Note that the macros for performing each of these steps take the form:

addBoundaryConditionVariable(<boundaryCondition>,<entityName>,<variableName>,<variableValue>)


These macros are documented at Euler Macro and Ideal MHD Macro. For the boundary condition on the inflowEntity, the steps are the same as we specified for the initial condition (i.e. the inflow boundary is held in the same state as at time $$t=0$$):

# Step 1: Add Variables

# Step 3: Add expressions specifying boundary condition on density,
# momentum, total energy


USim can compute additional quantities during the simulation that are of interest to the user. This is accomplised by the use of the macro:

addOutputDiagnostic(<outputDiagnosticName>)


This macro is documented at Euler Macro and Ideal MHD Macro. Once an output diagnostic has been defined, USim allows the user to compute the diagnostic using the familiar three-step process:

1. Use addOutputDiagnosticVariable to add variables that are independent of grid position.
2. Use addOutputDiagnosticPreExpression to add quantities that are functions of grid position, variables and any previously defined PreExpression in this block. Evaluated before expressions and the result is not accessible outside of this block. Any number of PreExpressions can be added.
3. Use addOutputDiagnosticExpression to compute the diagnostic.

Note that the macros for performing each of these steps take the form:

addOutputDiagnosticVariable(<outputDiagnosticName>,<variableName>,<variableValue>)


These macros are documented at Euler Macro and Ideal MHD Macro. In the Flow Over a Forward Facing Step example, we compute the mach number ($$M = \frac{|u|}{c_s}$$) as follows:

#  Add a diagnostic to compute the Mach number of the flow


Note

For the Euler equations, the following variables are pre-defined when computing an output diagnostic:

• rho
• rhoVx
• rhoVy
• rhoVz
• En
• Vx
• Vy
• Vz
• P

For the MHD equations, the following variables are pre-defined when computing an output diagnostic:

• rho
• rhoVx
• rhoVy
• rhoVz
• En
• Vx
• Vy
• Vz
• P
• Pb
• divB
• Jx
• Jy
• Jz

## Putting it all Together¶

The final step in the USim simulation is to add:

runFluidSimulation()


This tells USim that we’re done specifying the simulation and that it can be run. So, our simulation now looks like:

# Are we solving the MHD equations?
$MHD = False # Import macros to setup simulation$ import fluidsBase.mac
$if MHD$ import idealmhd.mac
$else$ import euler.mac
$endif # Specify parameters for the specific physics problem # grid for simulation$ GRIDFILE = "forwardFacingStep"
# format of grid
$GRIDTYPE = "gmsh" # adiabatic index$ GAS_GAMMA = 1.4
# Magnetic field strength
$BETA = 1.0e3 # end time for simulation$ TEND = 4.0
# number of frames
$NUMDUMPS = 20 # Riemann solver$ DIFFUSIVE = True
# Order in time
$TIME_ORDER = "second" # Write data for restarting the simulation$ WRITE_RESTART = False
# Output info for debugging purposes
$DEBUG = False # Dimensionality$ NDIM = 2
# Permeability of free space
$MU0 = 1.0 # CFL condition$ CFL = 0.4

# Initialize a USim simulation
$if MHD initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,MU0,WRITE_RESTART,DEBUG)$ else
initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,WRITE_RESTART,DEBUG)
$endif$ if isEqualString(GRIDTYPE,ExodusII)
$else addGmshGrid(GRIDFILE)$ endif

# Create data structures needed for the simulation
createFluidSimulation()

# Step 3: Add expressions specifying initial condition on density,
# momentum, total energy

# Create boundary entities
# Inflow for x < 0.0

# Wall for 0.0 < x < 3.0

# Outflow for x > 3.0

# Add the spatial discretization of the fluxes
finiteVolumeScheme(DIFFUSIVE)

# Boundary conditions
boundaryCondition(wall,wallEntity)
boundaryCondition(userSpecified,inflowEntity)
boundaryCondition(copy,outflowEntity)

# Specify the inflow boundary condition according to the problem initial conditions

#  Expressions to specify the inflow boundary

#  Add a diagnostic to compute the Mach number of the flow

# Time integration

# Run the simulation!
runFluidSimulation()


We can now apply our same approach to generalize this to performing simulations on an unstructured mesh in USim:

# Are we solving the MHD equations?
$MHD = True # Import macros to setup simulation$ import fluidsBase.mac
$if MHD$ import idealmhd.mac
$else$ import euler.mac
$endif # Specify parameters for the specific physics problem$ PARAM_1 = <value>
$PARAM_2 = <value>$ PARAM_N = <value>

# Initialize a USim simulation
$if MHD initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,MU0,WRITE_RESTART,DEBUG)$ else
initializeFluidSimulation(NDIM,0.0,TEND,NUMDUMPS,CFL,GAS_GAMMA,WRITE_RESTART,DEBUG)
$endif # Setup the grid$ if isEqualString(GRIDTYPE,ExodusII)
$else addGmshGrid(GRIDFILE)$ endif

# Create data structures needed for the simulation
createFluidSimulation()

# Specify initial condition

# Step 3: a) Add expressions specifying initial condition on density,
# momentum
$if MHD # Step 3: b) Add expression specifying initial conditions on total # energy, magnetic field, correction potential addExpression(<expression>)$ else
# Step 3: b) Add expression specifying initial conditions on total
# energy
\$ endif

# Define boundary entities

# Add the spatial discretization of the fluxes
finiteVolumeScheme(DIFFUSIVE)

# Boundary conditions
boundaryCondition(<boundaryCondition,entity>)

# User-specified boundary conditions
boundaryCondition(userSpecified, <entityName>)

# User-specified output diagnostics

# Time integration