seisflows.solver.specfem ======================== .. py:module:: seisflows.solver.specfem .. autoapi-nested-parse:: This Solver module is in charge of interacting with external numerical solvers such as SPECFEM (2D/3D/3D_GLOBE). This SPECFEM base class provides general functions that work with all versions of SPECFEM. Subclasses will provide additional capabilities unique to each version of SPECFEM. .. note:: The Base class implementation is almost completely SPECFEM2D related. However, SPECFEM2D requires a few unique parameters that 3D/3D_GLOBE do not. Because of the inheritance architecture of SeisFlows, we do not want the 3D and 3D_GLOBE versions to inherit 2D-specific parameters, so we need this this more generalized SPECFEM base class. TODO - add in `apply_hess` functionality that was partially written in legacy code - move `_initialize_adjoint_traces` to workflow.migration - Add density scaling based on Vp? Classes ------- .. autoapisummary:: seisflows.solver.specfem.Specfem Module Contents --------------- .. py:class:: Specfem(syn_data_format='ascii', materials='acoustic', update_density=False, nproc=1, ntask=1, attenuation=False, smooth_h=0.0, smooth_v=0.0, smooth_type='gaussian', components=None, source_prefix=None, mpiexec=None, workdir=os.getcwd(), path_solver=None, path_eval_grad=None, path_data=None, path_specfem_bin=None, path_specfem_data=None, path_model_init=None, path_model_true=None, path_output=None, **kwargs) Solver SPECFEM [Solver Base] ---------------------------- Defines foundational structure for Specfem-based solver module. Generalized SPECFEM interface to manipulate SPECFEM2D/3D/3D_GLOBE w/ Python Parameters ---------- :type syn_data_format: str :param syn_data_format: data format for reading synthetic traces into memory. Available: ['SU': seismic unix format, 'ASCII': human-readable ascii] :type materials: str or list :param materials: Material name used to define the model parameters that will be updated during an Inversion workflow. Available options (case-insensitive): - `type: list` (2D, 3D, 3D_GLOBE): User-defined list of lower-case parameters (e.g., ['vp', 'vs'] to mimic 'ELASTIC') NOTE: User is responsible for understanding if their chosen parameters are actually represented in SPECFEM, there are no guard rails here to protect incorrect parameter naming - ACOUSTIC (2D, 3D, 3D_GLOBE): vp - ELASTIC (2D, 3D, 3D_GLOBE): vp, vs - TRANSVERSE_ISOTROPIC (3D, 3D_GLOBE): vpv, vph, vsv, vsh, eta - 2D_ANISOTROPIC (2D): c11 c13 c15 c33 c35 c55 c12 c23 c25 c22 - ANISOTROPIC (3D, 3D_GLOBE): c_ij (21 parameter anisotropy) :type update_density: bool :param update_density: How to treat density during inversion. If True, updates density during inversion. If False, keeps it constant. TODO allow density scaling during an inversion :type attenuation: bool :param attenuation: How to treat attenuation during inversion. if True, turns on attenuation during forward simulations only. If False, attenuation is always set to False. Requires underlying attenution (Q_mu, Q_kappa) model :type smooth_h: float :param smooth_h: Gaussian half-width for horizontal smoothing in units of meters. If 0., no smoothing applied. Only applicable for workflows: ['migration', 'inversion'], ignored for 'forward' workflow. SPECFEM3D_GLOBE only: if `smooth_type`=='laplacian' then this is just the X and Y extent of the applied smoothing :type smooth_v: float :param smooth_v: Gaussian half-width for vertical smoothing in units of meters. Only applicable for workflows: ['migration', 'inversion'], ignored for 'forward' workflow. SPECFEM3D_GLOBE only: if `smooth_type`=='laplacian' then this is just the Z extent of the applied smoothing :type smooth_type: str :param smooth_type: choose how smoothing is performed for gradients. these are tied to the internal smoothing functions available, and only certain code flavors have certain smoothing functions available: - 'gaussian' [2D/3D/3D_GLOBE]: Default, convolve with a 3D gaussian, slow and computationally intensive. Only option for 2D. - 'laplacian' [3D_GLOBE]: RECOMMENDED FOR 3D_GLOBE. Average points around vertex to smooth. Much faster than Gaussian. - 'pde' [3D]: RECOMMENDED for 3D. Diffusion-based PDE smoothing. Much faster than Gaussian. See SPECFEM3D PR#1725 :type components: str :param components: components to search for synthetic data with. None by default which uses a wildcard when searching for synthetics. If provided, User only wants to use a subset of components generated by SPECFEM. In that case, `components` should be string of letters such as 'ZN' (for up and north components) :type solver_io: str :param solver_io: format of model/kernel/gradient files expected by the numerical solver. Available: ['fortran_binary': default .bin files]. TODO: ['adios': ADIOS formatted files] :type source_prefix: str :param source_prefix: prefix of source/event/earthquake files. If None, will attempt to guess based on the specific solver chosen. :type mpiexec: str :param mpiexec: MPI executable used to run parallel processes. Should also be defined for the system module Paths ----- :type path_data: str :param path_data: path to any externally stored waveform data required for data-synthetic comparison :type path_specfem_bin: str :param path_specfem_bin: path to SPECFEM bin/ directory which contains binary executables for running SPECFEM :type path_specfem_data: str :param path_specfem_data: path to SPECFEM DATA/ directory which must contain the CMTSOLUTION, STATIONS and Par_file files used for running SPECFEM *** .. py:attribute:: syn_data_format :value: 'ascii' .. py:attribute:: materials :value: 'acoustic' .. py:attribute:: nproc :value: 1 .. py:attribute:: ntask :value: 1 .. py:attribute:: update_density :value: False .. py:attribute:: attenuation :value: False .. py:attribute:: smooth_h :value: 0.0 .. py:attribute:: smooth_v :value: 0.0 .. py:attribute:: smooth_type :value: 'gaussian' .. py:attribute:: components :value: None .. py:attribute:: source_prefix :value: 'SOURCE' .. py:attribute:: prune_scratch :value: None .. py:attribute:: path .. py:attribute:: _parameters :value: [] .. py:attribute:: _mpiexec :value: None .. py:attribute:: _source_names :value: None .. py:attribute:: _ext :value: '' .. py:attribute:: _available_model_types :value: ['gll'] .. py:attribute:: _available_materials :value: None .. py:attribute:: _syn_available_data_formats :value: ['ASCII', 'SU'] .. py:attribute:: _required_binaries :value: ['xspecfem2D', 'xmeshfem2D', 'xcombine_sem'] .. py:attribute:: _acceptable_source_prefixes :value: ['SOURCE', 'FORCE', 'FORCESOLUTION'] .. py:attribute:: _fwd_simulation_executables :value: ['bin/xmeshfem2D', 'bin/xspecfem2D'] .. py:attribute:: _adj_simulation_executables :value: ['bin/xspecfem2D'] .. py:attribute:: _absorb_wildcard :value: 'absorb_*_*' .. py:attribute:: _forward_array_wildcard :value: '' .. py:attribute:: _regions :value: None .. py:attribute:: _export_vtk :value: False .. py:method:: check() Checks parameter validity for SPECFEM input files and model parameters .. py:method:: setup() Prepares solver scratch directories for an impending workflow. Sets up directory structure expected by SPECFEM and copies or generates seismic data to be inverted or migrated. Exports INIT/STARTING and TRUE/TARGET models to disk (output/ dir.) .. py:method:: check_model_values(path) Convenience function to check parameter and model validity for chosen Solver model. Should be called by the Workflow module :type path: str :param path: path to model file(s) that should be in the format expected by the Model class (FORTRAN binary, ADIOS etc.) .. py:method:: set_parameters(keys, vals, file, delim, **kwargs) Public API that allows other modules modify solver-specific files with paths relative to the `cwd` attribute. Primarily used to modify locations or force vector direction for the generation of different kernels in noise workflows. Only works if file exists, otherwise raises FileNotFoundError Kwargs are passed to `seisflows.tools.specfem.setpar()` :type key: str :param key: case-insensitive key to match in par_file. must match EXACT :type val: str :param val: value to OVERWRITE to the given key :raises FileNotFoundError: if `file` does not exist within the solver's working directory .. py:property:: source_names Returns list of source names which should be stored in PAR.SPECFEM_DATA Source names are expected to match the following wildcard, 'PREFIX_*' where PREFIX is something like 'CMTSOLUTION' or 'FORCE' .. note:: Dependent on environment variable 'SEISFLOWS_TASKID' which is assigned by system.run() to each individually running process. :rtype: list :return: list of source names .. py:property:: source_name Returns name of source currently under consideration .. note:: Dependent on environment variable 'SEISFLOWS_TASKID' which is assigned by system.run() to each individually running process. :rtype: str :return: given source name for given task id .. py:property:: cwd Returns working directory currently in use by a running solver instance .. note:: Dependent on environment variable 'SEISFLOWS_TASKID' which is assigned by system.run() to each individually running process. :rtype: str :return: current solver working directory .. py:method:: data_wildcard(comp='?') Returns a wildcard identifier for synthetic data based on SPECFEM2D file naming schema. Allows formatting dcomponent e.g., when called by solver.data_filenames. Some example SPECFEM2D ASCII seismogram file names for reference: - AA.S000000.BXY.semd: Membrane wave displacement - AA.S000000.BXY.semp: Membrane wave pressure - AA.S000000.PRE.semp: P-SV pressure seismogram .. note:: SPECFEM3D/3D_GLOBE versions must overwrite this function :type comp: str :param comp: component formatter, defaults to wildcard '?' :rtype: str :return: wildcard identifier for channels .. py:method:: model_wildcard(par='*', kernel=False) Returns a wildcard identifier to search for models kernels generated by the solver. An example SPECFEM2D/3D kernel filename (in FORTRAN binary file format) is: 'proc000001_rho_kernel.bin' Whereas the corresponding model would be 'proc000001_rho.bin' Allows dynamically searching for specific files when renaming, moving or copying files. Also allows for different wildcard for 3D_GLOBE version :type par: str :param par: parameter formatter, defaults to wildcard '?' :type kernel: bool :param kernel: wildcarding a kernel file. If True, adds the 'kernel' tag. If not, assuming we are wildcarding for a model file :rtype: str :return: wildcard identifier for channels .. py:method:: data_filenames(choice='obs') Returns the filenames of SPECFEM2D data, either by the requested components or by all available files in the directory. .. note:: SPECFEM3D/3D_GLOBE versions must overwrite this function .. note:: If the glob returns an empty list, this function exits the workflow because filenames should not be empty is they're being queried :rtype: list :return: list of data filenames .. py:property:: model_databases The location of model inputs and outputs as defined by SPECFEM2D. This is RELATIVE to a SPECFEM2D working directory. .. note:: This path is SPECFEM version dependent so SPECFEM3D/3D_GLOBE versions must overwrite this function :rtype: str :return: path where SPECFEM2D database files are stored, relative to `solver.cwd` .. py:property:: model_files Return a list of paths to model files that match the internal parameter list. Used to generate model vectors of the same length as gradients. :rtype: list :return: a list of full paths to model files that matches the internal list of solver parameters .. py:property:: kernel_databases The location of kernel inputs and outputs as defined by SPECFEM2D This is RELATIVE to a SPECFEM2D working directory. .. note:: This path is SPECFEM version dependent so SPECFEM3D/3D_GLOBE versions must overwrite this function :rtype: str :return: path where SPECFEM2D database files are stored, relative to `solver.cwd` .. py:method:: forward_simulation(save_traces=False, export_traces=False, save_forward_arrays=False, flag_save_forward=True, **kwargs) Wrapper for SPECFEM binaries: 'xmeshfem?D' 'xgenerate_databases', 'xspecfem?D' Calls SPECFEM2D forward solver, exports solver outputs to traces dir .. note:: SPECFEM3D/3D_GLOBE versions must overwrite this function :type save_traces: str :param save_traces: move files from their native SPECFEM output location to another directory. This is used to move output waveforms to 'traces/obs' or 'traces/syn' so that SeisFlows knows where to look for them, and so that SPECFEM doesn't overwrite existing files during subsequent forward simulations :type export_traces: str :param export_traces: export traces from the scratch directory to a more permanent storage location. i.e., copy files from their original location :type save_forward_arrays: str :param save_forward_arrays: relative path (relative to /scratch/solver//) to move the forward arrays which are used for adjoint simulations. Mainly used for ambient noise adjoint tomography which requires multiple forward simulations prior to adjoint simulations, putting forward arrays at the risk of overwrite. Normal Users can leave this default. :type flag_save_forward: bool :param flag_save_forward: whether to turn on the flag for saving the forward arrays which are used for adjoint simulations. Not required if only running forward simulations. .. py:method:: adjoint_simulation(save_kernels=False, export_kernels=False, load_forward_arrays=False, del_loaded_forward_arrays=False, **kwargs) Wrapper for SPECFEM binary 'xspecfem?D' Calls SPECFEM2D adjoint solver, creates the `SEM` folder with adjoint traces which is required by the adjoint solver. Renames kernels after they have been created from 'alpha' and 'beta' to 'vp' and 'vs', respectively. .. note:: SPECFEM3D/3D_GLOBE versions must overwrite this function :type save_kernels: str :param save_kernels: move the kernels from their native SPECFEM output location to another path. This is used to move kernels to another SeisFlows scratch directory so that they are discoverable by other modules. The typical location they are moved to is path_eval_grad :type export_kernels: str :param export_kernels: export/copy/save kernels from the scratch directory to a more permanent storage location. i.e., copy files from their original location. Note that kernel file sizes are LARGE, so exporting kernels can lead to massive storage requirements. :type load_forward_arrays: str :param load_forward_arrays: relative path (relative to solver.cwd) to load previously generated forward arrays which are used for adjoint simulations. Mainly used for ambient noise adjoint tomography. Will OVERWRITE any forward array files already located in the database directory. :type del_loaded_forward_arrays: bool :param del_loaded_forward_arrays: only used if `load_forward_arrays` is set. After adjoint simulation completes nominally, delete the forward arrays that were used to run the adjoint simulation to save space. Usually .. py:method:: _rename_kernel_parameters() Rename kernels to work w/ conflicting name conventions. - alpha -> vp - beta -> vs Performed directly inside the directory so that rename won't affect any strings in the full path. Deals with both SPECFEM3D and 3D_GLOBE. GLOBE version adds in the 'reg?' tag that needs to be considered. Kept as a separate function so it can be called outside the adjoint simulation task for debugging purposes. .. py:method:: combine(input_paths, output_path, parameters=None) Wrapper for 'xcombine_sem'. Sums kernels from individual source contributions to create gradient. .. note:: The binary xcombine_sem simply sums matching databases .. note:: It is ASSUMED that this function is being called by system.run(single=True) so that we can use the main solver directory to perform the kernel summation task :type input_paths: list :param input_paths: list of paths to directories containing binary files to be combined :type output_path: str :param output_path: path to export the outputs of xcombine_sem :type parameters: list :param parameters: optional list of parameters, defaults to `self._parameters` .. py:method:: smooth(input_path, output_path, parameters=None, span_h=None, span_v=None, use_gpu=False) Wrapper for SPECFEM smoothing binaries: xsmooth_sem, xsmooth_sem_pde, xsmooth_laplacian_sem User chooses which underlying function they want with the `smooth_type` parameter .. note:: It is ASSUMED that this function is being called by system.run(single=True) so that we can use the main solver directory to perform the kernel smooth task :type input_path: str :param input_path: path to data :type output_path: str :param output_path: path to export the outputs of xcombine_sem :type parameters: list :param parameters: optional list of parameters, defaults to `self._parameters` :type span_h: float :param span_h: horizontal smoothing length in meters :type span_v: float :param span_v: vertical smoothing length in meters :type use_gpu: bool :param use_gpu: whether to use GPU acceleration for smoothing. Requires GPU compiled binaries and GPU compute node. .. py:method:: _run_binary(executable, stdout='solver.log', with_mpi=True) Calls MPI solver executable to run solver binaries, used by individual processes to run the solver on system. If the external solver returns a non-zero exit code (failure), this function will return a negative boolean. .. note:: This function ASSUMES it is being run from a SPECFEM working directory, i.e., that the executables are located in ./bin/ .. note:: This is essentially an error-catching wrapper of subprocess.run() :type executable: str :param executable: executable function to call. May or may not start E.g., acceptable calls for the solver would './bin/xspecfem2D'. Also accepts additional command line arguments such as: 'xcombine_sem alpha_kernel kernel_paths...' :type stdout: str :param stdout: where to redirect stdout :type with_mpi: bool :param with_mpi: If `mpiexec` is given, use MPI to run the executable. Some executables (e.g., combine_vol_data_vtk) must be run in serial so this flag allows them to turn off MPI running. :raises SystemExit: If external numerical solver return any failure code while running .. py:method:: _exc2log(exc) :staticmethod: Very simple conversion utility to get log file names based on binaries. e.g., binary 'xspecfem2D' will return 'solver'. Helps keep log file naming consistent and generalizable TODO add a check here to see if the log file exists, and then use `number_fid` to increment so that we keep all the output logs :type exc: str :param exc: specfem executable, e.g., xspecfem2D, xgenerate_databases :rtype: str :return: logfile name that matches executable name .. py:method:: import_model(path_model) Copy files from given `path_model` into the current working directory model database. Used for grabbing starting models (e.g., MODEL_INIT) and models that have been perturbed by the optimization library. :type path_model: str :param path_model: path to an existing starting model .. py:method:: _initialize_working_directories(max_workers=None) Serial or parallel task used to initialize working directories for each of the available sources :type max_workers: int :param max_workers: number of concurrent tasks to use when creating working directories. Defaults to using all available cores on the machine since this is a lightweight task .. py:method:: _initialize_working_directory(cwd=None) Creates scratch directory structure expected by SPECFEM (i.e., bin, DATA, OUTPUT_FILES). Copies executables (bin) and input data (DATA) directories, prepares simulation input files. Each directory will act as completely independent Specfem working dir. This allows for embarrassing parallelization while avoiding the need for intra-directory communications, at the cost of temporary disk space. .. note:: path to binary executables must be supplied by user as SeisFlows has no mechanism for automatically compiling from source code. :type cwd: str :param cwd: optional scratch working directory to intialize. If None, will set based on current running seisflows task (self.taskid) .. py:method:: _export_starting_models(parameters=None) Export the initial and target models to the SeisFlows output/ directory. These are not used for actual simulations, just to keep track of models that were used during the workflow. :type parameters: list :param parameters: list of parameters to export. If None, will default to `self._parameters` .. py:method:: make_output_vtk_files(input_path, output_path=None, parameters=None, hi_res=False, tag=None, kernel=False) A warpper on `combine_vol_data_vtk()` that automatically tries to generate .vtk files using the SPECFEM binary xcombine_vol_data_vtk, and rename the output files to not be so generic. Files will be stored in the `output_path` directory, and will be named based on the `tag` unless overwritten by the User. :type input_path: str :param input_path: path to database files to be summed. :type output_path: strs :param output_path: path to export the outputs of the binary :type parameters: list :param parameters: optional list of parameters, defaults to `self._parameters` if None provided (e.g., ['vp', 'vs']) :type tag: str :param tag: optional tag to rename output vtk files. If not provided, will use the name of the directory holding the files :type kernel: bool :param kernel: whether the files being converted are kernel files or model files. This changes the file naming convention :type hi_res: bool :param hi_res: Set the high resolution flag to 1 or True, which will generate .vtk files with data at EACH GLL point, rather than at each nodal vertex. These files are LARGE, and we discourage using `hi_res`==True unless you know you want these files. .. py:method:: finalize() General finalization procedures for SPECFEM-based solver activities