seisflows.solver.specfem

This Solver module is in charge of interacting with external numerical solvers such as SPECFEM (2D/3D/3D_GLOBE). This SPECFEM base class provides general functions that work with all versions of SPECFEM. Subclasses will provide additional capabilities unique to each version of SPECFEM.

Note

The Base class implementation is almost completely SPECFEM2D related. However, SPECFEM2D requires a few unique parameters that 3D/3D_GLOBE do not. Because of the inheritance architecture of SeisFlows, we do not want the 3D and 3D_GLOBE versions to inherit 2D-specific parameters, so we need this this more generalized SPECFEM base class.

TODO

• add in apply_hess functionality that was partially written in legacy code

• move _initialize_adjoint_traces to workflow.migration

• Add density scaling based on Vp?

Classes

Specfem

Solver SPECFEM [Solver Base]

Module Contents

class seisflows.solver.specfem.Specfem(syn_data_format='ascii', materials='acoustic', update_density=False, nproc=1, ntask=1, attenuation=False, smooth_h=0.0, smooth_v=0.0, smooth_type='gaussian', components=None, source_prefix=None, mpiexec=None, workdir=os.getcwd(), path_solver=None, path_eval_grad=None, path_data=None, path_specfem_bin=None, path_specfem_data=None, path_model_init=None, path_model_true=None, path_output=None, **kwargs)

Solver SPECFEM [Solver Base]

Defines foundational structure for Specfem-based solver module. Generalized SPECFEM interface to manipulate SPECFEM2D/3D/3D_GLOBE w/ Python

Parameters

type syn_data_format:

str

param syn_data_format:

data format for reading synthetic traces into memory. Available: [‘SU’: seismic unix format, ‘ASCII’: human-readable ascii]

type materials:

str or list

param materials:

Material name used to define the model parameters that will be updated during an Inversion workflow. Available options (case-insensitive): - type: list (2D, 3D, 3D_GLOBE): User-defined list of lower-case

parameters (e.g., [‘vp’, ‘vs’] to mimic ‘ELASTIC’) NOTE: User is responsible for understanding if their chosen parameters are actually represented in SPECFEM, there are no guard rails here to protect incorrect parameter naming

• ACOUSTIC (2D, 3D, 3D_GLOBE): vp

• ELASTIC (2D, 3D, 3D_GLOBE): vp, vs

• TRANSVERSE_ISOTROPIC (3D, 3D_GLOBE): vpv, vph, vsv, vsh, eta

• 2D_ANISOTROPIC (2D): c11 c13 c15 c33 c35 c55 c12 c23 c25 c22

• ANISOTROPIC (3D, 3D_GLOBE): c_ij (21 parameter anisotropy)

type update_density:

bool

param update_density:

How to treat density during inversion. If True, updates density during inversion. If False, keeps it constant. TODO allow density scaling during an inversion

type attenuation:

bool

param attenuation:

How to treat attenuation during inversion. if True, turns on attenuation during forward simulations only. If False, attenuation is always set to False. Requires underlying attenution (Q_mu, Q_kappa) model

type smooth_h:

float

param smooth_h:

Gaussian half-width for horizontal smoothing in units of meters. If 0., no smoothing applied. Only applicable for workflows: [‘migration’, ‘inversion’], ignored for ‘forward’ workflow. SPECFEM3D_GLOBE only: if `smooth_type`==’laplacian’ then this is just the X and Y extent of the applied smoothing

type smooth_v:

float

param smooth_v:

Gaussian half-width for vertical smoothing in units of meters. Only applicable for workflows: [‘migration’, ‘inversion’], ignored for ‘forward’ workflow. SPECFEM3D_GLOBE only: if `smooth_type`==’laplacian’ then this is just the Z extent of the applied smoothing

type smooth_type:

str

param smooth_type:

choose how smoothing is performed for gradients. these are tied to the internal smoothing functions available, and only certain code flavors have certain smoothing functions available: - ‘gaussian’ [2D/3D/3D_GLOBE]: Default, convolve with a 3D gaussian,

slow and computationally intensive. Only option for 2D.

• ‘laplacian’ [3D_GLOBE]: RECOMMENDED FOR 3D_GLOBE. Average points

  around vertex to smooth. Much faster than Gaussian.

• ‘pde’ [3D]: RECOMMENDED for 3D. Diffusion-based PDE smoothing.

  Much faster than Gaussian. See SPECFEM3D PR#1725

type components:

str

param components:

components to search for synthetic data with. None by default which uses a wildcard when searching for synthetics. If provided, User only wants to use a subset of components generated by SPECFEM. In that case, components should be string of letters such as ‘ZN’ (for up and north components)

type solver_io:

str

param solver_io:

format of model/kernel/gradient files expected by the numerical solver. Available: [‘fortran_binary’: default .bin files]. TODO: [‘adios’: ADIOS formatted files]

type source_prefix:

str

param source_prefix:

prefix of source/event/earthquake files. If None, will attempt to guess based on the specific solver chosen.

type mpiexec:

str

param mpiexec:

MPI executable used to run parallel processes. Should also be defined for the system module

Paths

type path_data:

str

param path_data:

path to any externally stored waveform data required for data-synthetic comparison

type path_specfem_bin:

str

param path_specfem_bin:

path to SPECFEM bin/ directory which contains binary executables for running SPECFEM

type path_specfem_data:

str

param path_specfem_data:

path to SPECFEM DATA/ directory which must contain the CMTSOLUTION, STATIONS and Par_file files used for running SPECFEM

***

syn_data_format = 'ascii'

materials = 'acoustic'

nproc = 1

ntask = 1

update_density = False

attenuation = False

smooth_h = 0.0

smooth_v = 0.0

smooth_type = 'gaussian'

components = None

source_prefix = 'SOURCE'

prune_scratch = None

path

_parameters = []

_mpiexec = None

_source_names = None

_ext = ''

_available_model_types = ['gll']

_available_materials = None

_syn_available_data_formats = ['ASCII', 'SU']

_required_binaries = ['xspecfem2D', 'xmeshfem2D', 'xcombine_sem']

_acceptable_source_prefixes = ['SOURCE', 'FORCE', 'FORCESOLUTION']

_fwd_simulation_executables = ['bin/xmeshfem2D', 'bin/xspecfem2D']

_adj_simulation_executables = ['bin/xspecfem2D']

_absorb_wildcard = 'absorb_*_*'

_forward_array_wildcard = ''

_regions = None

_export_vtk = False

check()

Checks parameter validity for SPECFEM input files and model parameters

setup()

Prepares solver scratch directories for an impending workflow.

Sets up directory structure expected by SPECFEM and copies or generates seismic data to be inverted or migrated.

Exports INIT/STARTING and TRUE/TARGET models to disk (output/ dir.)

check_model_values(path)

Convenience function to check parameter and model validity for chosen Solver model. Should be called by the Workflow module

Parameters:

path (str) – path to model file(s) that should be in the format expected by the Model class (FORTRAN binary, ADIOS etc.)

set_parameters(keys, vals, file, delim, **kwargs)

Public API that allows other modules modify solver-specific files with paths relative to the cwd attribute.

Primarily used to modify locations or force vector direction for the generation of different kernels in noise workflows.

Only works if file exists, otherwise raises FileNotFoundError Kwargs are passed to seisflows.tools.specfem.setpar()

Parameters:

• key (str) – case-insensitive key to match in par_file. must match EXACT

• val (str) – value to OVERWRITE to the given key

Raises:

FileNotFoundError – if file does not exist within the solver’s working directory

property source_names

Returns list of source names which should be stored in PAR.SPECFEM_DATA Source names are expected to match the following wildcard, ‘PREFIX_*’ where PREFIX is something like ‘CMTSOLUTION’ or ‘FORCE’

Note

Dependent on environment variable ‘SEISFLOWS_TASKID’ which is assigned by system.run() to each individually running process.

Return type:

list

Returns:

list of source names

property source_name

Returns name of source currently under consideration

Note

Dependent on environment variable ‘SEISFLOWS_TASKID’ which is assigned by system.run() to each individually running process.

Return type:

str

Returns:

given source name for given task id

property cwd

Returns working directory currently in use by a running solver instance

Note

Dependent on environment variable ‘SEISFLOWS_TASKID’ which is assigned by system.run() to each individually running process.

Return type:

str

Returns:

current solver working directory

data_wildcard(comp='?')

Returns a wildcard identifier for synthetic data based on SPECFEM2D file naming schema. Allows formatting dcomponent e.g., when called by solver.data_filenames.

Some example SPECFEM2D ASCII seismogram file names for reference: - AA.S000000.BXY.semd: Membrane wave displacement - AA.S000000.BXY.semp: Membrane wave pressure - AA.S000000.PRE.semp: P-SV pressure seismogram

Note

SPECFEM3D/3D_GLOBE versions must overwrite this function

Parameters:

comp (str) – component formatter, defaults to wildcard ‘?’

Return type:

str

Returns:

wildcard identifier for channels

model_wildcard(par='*', kernel=False)

Returns a wildcard identifier to search for models kernels generated by the solver. An example SPECFEM2D/3D kernel filename (in FORTRAN binary file format) is: ‘proc000001_rho_kernel.bin’ Whereas the corresponding model would be ‘proc000001_rho.bin’

Allows dynamically searching for specific files when renaming, moving or copying files. Also allows for different wildcard for 3D_GLOBE version

Parameters:

• par (str) – parameter formatter, defaults to wildcard ‘?’

• kernel (bool) – wildcarding a kernel file. If True, adds the ‘kernel’ tag. If not, assuming we are wildcarding for a model file

Return type:

str

Returns:

wildcard identifier for channels

data_filenames(choice='obs')

Returns the filenames of SPECFEM2D data, either by the requested components or by all available files in the directory.

Note

SPECFEM3D/3D_GLOBE versions must overwrite this function

Note

If the glob returns an empty list, this function exits the workflow because filenames should not be empty is they’re being queried

Return type:

list

Returns:

list of data filenames

property model_databases

The location of model inputs and outputs as defined by SPECFEM2D. This is RELATIVE to a SPECFEM2D working directory.

Note

This path is SPECFEM version dependent so SPECFEM3D/3D_GLOBE versions must overwrite this function

Return type:

str

Returns:

path where SPECFEM2D database files are stored, relative to solver.cwd

property model_files

Return a list of paths to model files that match the internal parameter list. Used to generate model vectors of the same length as gradients.

Return type:

list

Returns:

a list of full paths to model files that matches the internal list of solver parameters

property kernel_databases

The location of kernel inputs and outputs as defined by SPECFEM2D This is RELATIVE to a SPECFEM2D working directory.

Note

This path is SPECFEM version dependent so SPECFEM3D/3D_GLOBE versions must overwrite this function

Return type:

str

Returns:

path where SPECFEM2D database files are stored, relative to solver.cwd

forward_simulation(save_traces=False, export_traces=False, save_forward_arrays=False, flag_save_forward=True, **kwargs)

Wrapper for SPECFEM binaries: ‘xmeshfem?D’ ‘xgenerate_databases’,

‘xspecfem?D’

Calls SPECFEM2D forward solver, exports solver outputs to traces dir

Note

SPECFEM3D/3D_GLOBE versions must overwrite this function

Parameters:

• save_traces (str) – move files from their native SPECFEM output location to another directory. This is used to move output waveforms to ‘traces/obs’ or ‘traces/syn’ so that SeisFlows knows where to look for them, and so that SPECFEM doesn’t overwrite existing files during subsequent forward simulations

• export_traces (str) – export traces from the scratch directory to a more permanent storage location. i.e., copy files from their original location

• save_forward_arrays (str) – relative path (relative to /scratch/solver/<source_name>/<model_database>) to move the forward arrays which are used for adjoint simulations. Mainly used for ambient noise adjoint tomography which requires multiple forward simulations prior to adjoint simulations, putting forward arrays at the risk of overwrite. Normal Users can leave this default.

• flag_save_forward (bool) – whether to turn on the flag for saving the forward arrays which are used for adjoint simulations. Not required if only running forward simulations.

adjoint_simulation(save_kernels=False, export_kernels=False, load_forward_arrays=False, del_loaded_forward_arrays=False, **kwargs)

Wrapper for SPECFEM binary ‘xspecfem?D’

Calls SPECFEM2D adjoint solver, creates the SEM folder with adjoint traces which is required by the adjoint solver. Renames kernels after they have been created from ‘alpha’ and ‘beta’ to ‘vp’ and ‘vs’, respectively.

Note

SPECFEM3D/3D_GLOBE versions must overwrite this function

Parameters:

• save_kernels (str) – move the kernels from their native SPECFEM output location to another path. This is used to move kernels to another SeisFlows scratch directory so that they are discoverable by other modules. The typical location they are moved to is path_eval_grad

• export_kernels (str) – export/copy/save kernels from the scratch directory to a more permanent storage location. i.e., copy files from their original location. Note that kernel file sizes are LARGE, so exporting kernels can lead to massive storage requirements.

• load_forward_arrays (str) – relative path (relative to solver.cwd) to load previously generated forward arrays which are used for adjoint simulations. Mainly used for ambient noise adjoint tomography. Will OVERWRITE any forward array files already located in the database directory.

• del_loaded_forward_arrays (bool) – only used if load_forward_arrays is set. After adjoint simulation completes nominally, delete the forward arrays that were used to run the adjoint simulation to save space. Usually

_rename_kernel_parameters()

Rename kernels to work w/ conflicting name conventions. - alpha -> vp - beta -> vs

Performed directly inside the directory so that rename won’t affect any strings in the full path. Deals with both SPECFEM3D and 3D_GLOBE. GLOBE version adds in the ‘reg?’ tag that needs to be considered.

Kept as a separate function so it can be called outside the adjoint simulation task for debugging purposes.

combine(input_paths, output_path, parameters=None)

Wrapper for ‘xcombine_sem’. Sums kernels from individual source contributions to create gradient.

Note

The binary xcombine_sem simply sums matching databases

Note

It is ASSUMED that this function is being called by system.run(single=True) so that we can use the main solver directory to perform the kernel summation task

Parameters:

• input_paths (list) – list of paths to directories containing binary files to be combined

• output_path (str) – path to export the outputs of xcombine_sem

• parameters (list) – optional list of parameters, defaults to self._parameters

smooth(input_path, output_path, parameters=None, span_h=None, span_v=None, use_gpu=False)

Wrapper for SPECFEM smoothing binaries: xsmooth_sem, xsmooth_sem_pde, xsmooth_laplacian_sem User chooses which underlying function they want with the smooth_type parameter

Note

It is ASSUMED that this function is being called by system.run(single=True) so that we can use the main solver directory to perform the kernel smooth task

Parameters:

• input_path (str) – path to data

• output_path (str) – path to export the outputs of xcombine_sem

• parameters (list) – optional list of parameters, defaults to self._parameters

• span_h (float) – horizontal smoothing length in meters

• span_v (float) – vertical smoothing length in meters

• use_gpu (bool) – whether to use GPU acceleration for smoothing. Requires GPU compiled binaries and GPU compute node.

_run_binary(executable, stdout='solver.log', with_mpi=True)

Calls MPI solver executable to run solver binaries, used by individual processes to run the solver on system. If the external solver returns a non-zero exit code (failure), this function will return a negative boolean.

Note

This function ASSUMES it is being run from a SPECFEM working directory, i.e., that the executables are located in ./bin/

Note

This is essentially an error-catching wrapper of subprocess.run()

Parameters:

• executable (str) – executable function to call. May or may not start E.g., acceptable calls for the solver would ‘./bin/xspecfem2D’. Also accepts additional command line arguments such as: ‘xcombine_sem alpha_kernel kernel_paths…’

• stdout (str) – where to redirect stdout

• with_mpi (bool) – If mpiexec is given, use MPI to run the executable. Some executables (e.g., combine_vol_data_vtk) must be run in serial so this flag allows them to turn off MPI running.

Raises:

SystemExit – If external numerical solver return any failure code while running

static _exc2log(exc)

Very simple conversion utility to get log file names based on binaries. e.g., binary ‘xspecfem2D’ will return ‘solver’. Helps keep log file naming consistent and generalizable

TODO add a check here to see if the log file exists, and then use

number_fid to increment so that we keep all the output logs

Parameters:

exc (str) – specfem executable, e.g., xspecfem2D, xgenerate_databases

Return type:

str

Returns:

logfile name that matches executable name

import_model(path_model)

Copy files from given path_model into the current working directory model database. Used for grabbing starting models (e.g., MODEL_INIT) and models that have been perturbed by the optimization library.

Parameters:

path_model (str) – path to an existing starting model

_initialize_working_directories(max_workers=None)

Serial or parallel task used to initialize working directories for each of the available sources

Parameters:

max_workers (int) – number of concurrent tasks to use when creating working directories. Defaults to using all available cores on the machine since this is a lightweight task

_initialize_working_directory(cwd=None)

Creates scratch directory structure expected by SPECFEM (i.e., bin, DATA, OUTPUT_FILES). Copies executables (bin) and input data (DATA) directories, prepares simulation input files.

Each directory will act as completely independent Specfem working dir. This allows for embarrassing parallelization while avoiding the need for intra-directory communications, at the cost of temporary disk space.

Note

path to binary executables must be supplied by user as SeisFlows has no mechanism for automatically compiling from source code.

Parameters:

cwd (str) – optional scratch working directory to intialize. If None, will set based on current running seisflows task (self.taskid)

_export_starting_models(parameters=None)

Export the initial and target models to the SeisFlows output/ directory. These are not used for actual simulations, just to keep track of models that were used during the workflow.

Parameters:

parameters (list) – list of parameters to export. If None, will default to self._parameters

make_output_vtk_files(input_path, output_path=None, parameters=None, hi_res=False, tag=None, kernel=False)

A warpper on combine_vol_data_vtk() that automatically tries to generate .vtk files using the SPECFEM binary xcombine_vol_data_vtk, and rename the output files to not be so generic. Files will be stored in the output_path directory, and will be named based on the tag unless overwritten by the User.

Parameters:

• input_path (str) – path to database files to be summed.

• output_path (strs) – path to export the outputs of the binary

• parameters (list) – optional list of parameters, defaults to self._parameters if None provided (e.g., [‘vp’, ‘vs’])

• tag (str) – optional tag to rename output vtk files. If not provided, will use the name of the directory holding the files

• kernel (bool) – whether the files being converted are kernel files or model files. This changes the file naming convention

• hi_res (bool) – Set the high resolution flag to 1 or True, which will generate .vtk files with data at EACH GLL point, rather than at each nodal vertex. These files are LARGE, and we discourage using `hi_res`==True unless you know you want these files.

finalize()

General finalization procedures for SPECFEM-based solver activities