seisflows.tools.config ====================== .. py:module:: seisflows.tools.config .. autoapi-nested-parse:: Seisflows configuration tools, containing core utilities that are called upon throughout the Seisflows workflow. Attributes ---------- .. autoapisummary:: seisflows.tools.config.ENV_VARIABLES Classes ------- .. autoapisummary:: seisflows.tools.config.Dict seisflows.tools.config.Null Functions --------- .. autoapisummary:: seisflows.tools.config.load_yaml seisflows.tools.config.get_task_id seisflows.tools.config.set_task_id seisflows.tools.config.import_seisflows seisflows.tools.config.config_logger seisflows.tools.config.custom_import seisflows.tools.config.pickle_function_list seisflows.tools.config.copy_file seisflows.tools.config.number_fid Module Contents --------------- .. py:data:: ENV_VARIABLES :value: ['SEISFLOWS_TASKID', 'SLURM_ARRAY_TASK_ID'] .. py:class:: Dict Bases: :py:obj:`dict` A dictionary replacement which allows for easier parameter access through getting and setting attributes. Also has some functionality to make string printing prettier .. py:method:: __str__() Pretty print dictionaries and first level nested dictionaries .. py:method:: __repr__() Pretty print when calling an instance of this object .. py:method:: __getattr__(key) Attribute-like access of the internal dictionary attributes .. py:method:: __setattr__(key, val) Setting attributes can only be performed one time .. py:class:: Null(*args, **kwargs) A null object that always and reliably does nothing .. py:method:: __call__(*args, **kwargs) .. py:method:: __bool__() .. py:method:: __nonzero__() .. py:method:: __getattr__(key) .. py:method:: __setattr__(key, val) .. py:method:: __delattr__(key) .. py:function:: load_yaml(filename) Define how the PyYaml yaml loading function behaves. Replaces None and inf strings with NoneType and numpy.inf respectively Also expands all paths (parameters that start with 'path_') to be absolute :type filename: str :param filename: .yaml file to load in :rtype: Dict :return: Dictionary containing all parameters in a YAML file .. py:function:: get_task_id() Task IDs are assigned to each child process spawned by the system module during a SeisFlows workflow. SeisFlows modules use this Task ID to keep track of embarassingly parallel process, e.g., solver uses the Task ID to determine which source is being considered. :rtype: int :return: task id for given solver .. py:function:: set_task_id(task_id) Set the SEISFLOWS_TASKID in os environs for local workflows. If running on HPC systems, running array jobs will assign the Task ID .. note:: Mostly used for debugging/testing purposes as a way of mimicing system.run() assigning task ids to child processes :type task_id: int :param task_id: integer task id to assign to the current working environment .. py:function:: import_seisflows(workdir=os.getcwd(), parameter_file='parameters.yaml', **kwargs) Critical SeisFlows workflow setup block which runs a number of setup tasks prior to starting a workflow, tasks include: - loading the user-defined parameter file - configure the package-wide logger based on user-input path and verbosity - instantiate user-selected modules with the appropriate parameters - assign all non-workflow modules as attributes of the `workflow` module - return `workflow` module which now has access to all parameters and all modules and can be used to run SeisFlows :type workdir: str :param workdir: the current working directory in which to perform a SeisFlows workflow. Defaults to the current working directory :type parameter_file: str :param parameter_file: the YAML formatted parameter file that is used to instantiate each of the SeisFlows modules and run the workflow. This should be created by the command line argument 'seisflows configure'. Defaults to 'parameters.yaml' :rtype: module :return: instantiated 'workflow' module which contains all sub-modules which have been instantiated with user-defined parameters .. py:function:: config_logger(level='DEBUG', filename=None, filemode='a', verbose=True, stream_handler=True) Explicitely configure the logging module with some parameters defined by the user in the System module. Instantiates a stream logger to write to stdout, and a file logger which writes to `filename`. Two levels of verbosity and three levels of log messages allow the user to determine how much output they want to see. :type level: str :param level: log level to be passed to logger, available are 'CRITICAL', 'WARNING', 'INFO', 'DEBUG' :type filename: str or None :param filename: name of the log file to write log statements to. If None, logs will be written to STDOUT ONLY, and `filemode` will not be used. :type filemode: str :param filemode: method for opening the log file. defaults to append 'a' :type verbose: bool :param verbose: if True, writes a more detailed log message stating the type of log (warning, info, debug), and the class and method which called the logger (e.g., seisflows.solver.specfem2d.save()). This is much more useful for debugging but clutters up the log file. if False, only write the time and message in the log statement. .. py:function:: custom_import(name=None, module=None, classname=None) Imports SeisFlows module and extracts class that is the camelcase version of the module name. Used to dynamically import sub-modules by name only, avoiding the need to hardcode import statements. For example: custom_import('workflow', 'inversion') imports 'seisflows.workflow.inversion' and, from this module, extracts class 'Inversion'. :type name: str :param name: component of the workflow to import, defined by `names`, available: "system", "preprocess", "solver", "postprocess", "optimize", "workflow" :type module: module within the workflow component to call upon, e.g. seisflows.workflow.inversion, where `inversion` is the module :type classname: str :param classname: the class to be called from the module. Usually this is just the CamelCase version of the module, which will be defaulted to if this parameter is set `None`, however allows for custom class naming. Note: CamelCase class names following PEP-8 convention. .. py:function:: pickle_function_list(functions, path=os.getcwd(), **kwargs) Save a list of functions and their keyword arguments as pickle files. Return the names of the files. Used for running functions from spawned processes during cluster runs. .. note:: The idea here is that we need this list of functions to be discoverable by a system separate to the one that defined them. To do this we can pickle Python objects on disk, and have the new system read in the pickle files and evaluate the objects. We use 'dill' because Pickle can't serialize methods/functions :type functions: list of methods :param functions: a list of functions that should be run in order. All kwargs passed to run() will be passed into the functions. :type path: str :param path: path to save the pickle files. Defaults to current working directory :rtype: tuple of str :return: (name of the pickle file containing the function, name of the pickle file containing keyword arguments) .. py:function:: copy_file(fid, copy_to='./') Copy files to a location but do not overwrite if file already exists in `copy_to`. If the file already exists, append a number before the file ext. This is mainly used when resuming a workflow, where old log and parameter files may be overwritten or changed. :type fid: str :param fid: full path to the file that should be copied. If `fid` does not exist, nothing happens :type copy_to: str :param copy_to: location to copy the file to. file name will be the same. defaults to current working directory .. py:function:: number_fid(fid, i=0) Number a filename. Used to store old log files without overwriting them. Premise is, if you have a file e.g., called: output.txt This function would return incrementing filenames: output_000.txt, output_001.txt, output_002.txt, ouput_003.txt ... .. note:: Replace statement is catch-all, so we assume that there is only one instance of the file extension in the entire path. :type fid: str :param fid: path to the file that you want to increment :type i: int :param i: number to append to file id :rtype: str :return: filename with appended number. filename ONLY, will strip away the original path location