Troubleshooting Crashes During Stepping of Particle Simulations
Once the simulation starts timestepping, most potential causes of premature termination of the simulation have been passed. If the simulation does end prematurely, there are several steps we can take to diagnose what is happening and speed up resolving whatever issues are causing these.
Firstly, there are several potential causes for crashes:
The simulation is taking up too much memory.
The simulation is trying to access areas of memory it should not.
The simulation is looking for an object which it cannot find.
The hard drive has run out of space.
If the simulation is taking up too much memory, it is a good idea to
determine if the number of particles in the simulation is rising
steadily or running away. Instrumenting a history to measure the
number of macroparticles in each species is a good way to determine if
this is the cause. Rerun the simulation to a few steps short of where
the crash happened previously, then reduce the dumpPeriodicity
to get more information about the steps where the crash is
occuring. If the crash is due to a runaway of the number of macros
inside the simulation volume, then it will often be preceeded by a
slowing down of the wall-clock time per step.
If the number of particles is reasonable, then it is possible for particles to be finding their way out of the simulation domain. All particle simulations should include an absorbing box on the perimiter of the simulation. This is automatic in visual setup simulations, and is the responsibility of the user in text setup examples. In some rare cases there is an interaction between a separate absorber on a geometry and the perimiter abosrber that can lead to crashes due to particles getting outside the domain. The simulation is trying to determine a field at a location that is not in memory. This is very rare and should be reported to support@txcorp.com. This can also happen if the particles are being accelerated out of the domain. In these cases, the increase of dump periodicity is useful, but so also is the computePtclLimits analyzer, which will find particles at highest x, lowest x, highest velocity, and report that information in a history. If this shows particles exiting the simulation, then the input file(s) can be adjusted to avoid the possibility that the particles might escape there.
Other situations where a crash may occur later in a simulation include, where a secondary emitter is poorly defined, and it is some time before the shape on which that emission occurs is struck by a particle that can trigger this process.