biolockj.module

Interface BioModule

All Known Subinterfaces:

ApiModule, ClassifierModule, DatabaseModule, JavaModule, OutsidePipelineWriter, ParserModule, ReadCounter, ScriptModule, SeqModule

All Known Implementing Classes:

AddMetadataToPathwayTables, AddMetadataToTaxaTables, AddPseudoCount, AwkFastaConverter, BioModuleImpl, BuildQiimeMapping, BuildTaxaTables, ClassifierModuleImpl, CompileOtuCounts, Demultiplexer, DESeq2, EdgeR, Email, ForEachFile, ForEachLevel, ForEachLoop, ForEachSample, GenMod, GenomeAssembly, Gunzipper, Hello_Friends, Hello_World, Humann2Classifier, Humann2CountModule, Humann2Parser, ImportMetadata, JavaModuleImpl, JsonReport, KneadData, Kraken2Classifier, Kraken2Parser, KrakenClassifier, KrakenParser, LogTransformTaxaTables, MergeQiimeOtuTables, Metaphlan2Classifier, Metaphlan2Parser, Multiplexer, NormalizeByReadsPerMillion, NormalizeTaxaTables, OtuCountModule, ParserModuleImpl, PearMergeReads, QiimeClassifier, QiimeClosedRefClassifier, QiimeDeNovoClassifier, QiimeOpenRefClassifier, QiimeParser, R_CalculateStats, R_Module, R_PlotEffectSize, R_PlotMds, R_PlotOtus, R_PlotPvalHistograms, RarefyOtuCounts, RarefySeqs, RdpClassifier, RdpHierParser, RdpParser, RegisterNumReads, RemoveLowOtuCounts, RemoveLowPathwayCounts, RemoveScarceOtuCounts, RemoveScarcePathwayCounts, Rmarkdown, ScriptModuleImpl, SeqFileValidator, SeqModuleImpl, SequenceReadArchive, ShannonDiversity, SraDownload, SraMetaData, SraMetaDB, SrpSrrConverter, Step1, Step2, Step3, Step4, Step5, Stop, TaxaCountModule, TransformTaxaTables, TrimPrimers

public interface BioModule

Classes that implement this interface are eligible to be included in a BioLockJ pipeline.
Use the #BioModule tag with the class name in the Config file to include a module.
The Pipeline class executes BioModules in the order provided in the Config file.

BioModule Directory Structure

Directory	Description
output	Contains all module output files
temp	Holds intermediate files generated by the module, but are not to be passed on to the next module. This directory will deleted after pipeline execution if "pipeline.deleteTempFiles" = "Y"

Field Summary

Fields

Modifier and Type	Field and Description
static String	LOG_DIR Name of the temporary sub-directory: "log"
static String	MAIN_SCRIPT_PREFIX Script prefix appended to start of file name to indicate the main script in the script directory. Non-AWS pipelines execute worker scripts via executing the main shell script - named with the prefix: "MAIN_"
static String	NO_VERSION
static String	OUTPUT_DIR Name of the output sub-directory: "output"
static String	RES_DIR Name of the temporary sub-directory: "resources"
static String	TEMP_DIR Name of the temporary sub-directory: "temp"

Method Summary

Modifier and Type	Method and Description
void	checkDependencies() During pipeline initialization, all configured BioModules will run this method to validate dependencies.
void	cleanUp() This method executes after execution to update Config modified by the module or other cleanup operations.
void	executeTask() This is the main method called when it is time for the BioModule to complete its task.
String	getAlias() Some BioModules may be added to a pipeline multiple times. The user may provide an alias for a module in the run order, thus allowing the user direct properties to an individual instance of a module.
String	getDockerImageName() Get the docker image to use for this module.
String	getDockerImageOwner() Get the name of the docker hub user that owns the image to use for this module.
String	getDockerImageTag() Get the version / tag to use for the docker image.
Integer	getID() Some BioModules may be added to a pipeline multiple times so must be identified by an ID. This is the same value as the directory folder prefix when run. The 1st module ID is 0 (or 00 if there are more than 10 modules.
List<File>	getInputFiles() Each BioModule takes the previous BioModule output as input: BioModule[ n ].getInputFiles() = BioModule[ n - 1 ].getOutputDir().listFiles() Special cases: The 1st BioModule return all files in "input.dirPaths" If previous BioModule BioModule[ n - 1 ] is a MetadataModule, forward it's input + output file: BioModule[ n ].getInputFiles() = BioModule[ n -1 ].getInputFiles() + BioModule[ n -1 ].getOutputFiles()
File	getLogDir() Retains records of the process of running the module. The files are intended to be small and stored long term with successful pipelines.
File	getMetadata() Updated/new metadata files are saved to the module output directory (if created by the module).
File	getModuleDir() Each BioModule generates sub-directory under $DOCKER_PROJ
File	getOutputDir() Output files destined as input for the next BioModule is created in this directory.
List<String>	getPostRequisiteModules() Pipeline calls this method when building the list of pipeline BioModules to execute.
List<String>	getPreRequisiteModules() Pipeline calls this method when building the list of pipeline BioModules to execute.
String	getPropDefault(String prop) If a property is null based on the config files (including all defaults and standard.properties) but a module is passed to the Config class as the context for getting that module, the Config class can query the module for a value for the property.
File	getResourceDir() The resource sub-direcotry contains files that the module may depend on. Unlike temp or log, these files are not created by the module.
String	getSummary() Gets the BioModule execution summary, this is sent as part of the notification email, if configured. Summary should not include data content, to avoid unintentional publication of confidential information. However, meta-data such as number/size of files can be helpful during debug.
File	getTempDir() Contains intermediate files generated by the module but not used by the next BioModule. The files may contain supplementary information or data that may be helpful during debug or recovery. If "pipeline.deleteTempFiles" = "Y", successful pipelines delete this directory.
void	init() Initialize a new module to generate a unique ID and module directory.
boolean	isValidInputModule(BioModule previousModule) BioModules getInputFiles() method typically, but not always, return the previousModule output files.
void	setAlias(String alias)
default String	version() Changes to a module class should be accompanied by a increment in version.

Field Detail

MAIN_SCRIPT_PREFIX

static final String MAIN_SCRIPT_PREFIX

Script prefix appended to start of file name to indicate the main script in the script directory.
Non-AWS pipelines execute worker scripts via executing the main shell script - named with the prefix: "MAIN_"

See Also:

Constant Field Values

OUTPUT_DIR

static final String OUTPUT_DIR

Name of the output sub-directory: "output"

See Also:

Constant Field Values

TEMP_DIR

static final String TEMP_DIR

Name of the temporary sub-directory: "temp"

See Also:

Constant Field Values

LOG_DIR

static final String LOG_DIR

Name of the temporary sub-directory: "log"

See Also:

Constant Field Values

RES_DIR

static final String RES_DIR

Name of the temporary sub-directory: "resources"

See Also:

Constant Field Values

NO_VERSION

static final String NO_VERSION

See Also:

Constant Field Values

Method Detail

checkDependencies

void checkDependencies()
                throws Exception

During pipeline initialization, all configured BioModules will run this method to validate dependencies.

Throws:

Exception - thrown if missing or invalid dependencies are found

cleanUp

void cleanUp()
      throws Exception

This method executes after execution to update Config modified by the module or other cleanup operations.

Throws:

Exception - thrown if any runtime error occurs

executeTask

void executeTask()
          throws Exception

This is the main method called when it is time for the BioModule to complete its task.

Throws:

Exception - thrown if the module is unable to complete is task

getID

Integer getID()

Some BioModules may be added to a pipeline multiple times so must be identified by an ID.
This is the same value as the directory folder prefix when run.
The 1st module ID is 0 (or 00 if there are more than 10 modules.

Returns:

Module ID

getAlias

String getAlias()

Some BioModules may be added to a pipeline multiple times.
The user may provide an alias for a module in the run order, thus allowing the user direct properties to an individual instance of a module.

Returns:

Module ID

setAlias

void setAlias(String alias)
       throws PipelineFormationException

Throws:

PipelineFormationException

getInputFiles

List<File> getInputFiles()

Each BioModule takes the previous BioModule output as input:
BioModule[ n ].getInputFiles() = BioModule[ n - 1 ].getOutputDir().listFiles()
Special cases:

• The 1st BioModule return all files in "input.dirPaths"
  
• If previous BioModule BioModule[ n - 1 ] is a MetadataModule, forward it's input + output file: BioModule[ n ].getInputFiles() = BioModule[ n -1 ].getInputFiles() + BioModule[ n -1 ].getOutputFiles()

Returns:

Input files

getMetadata

File getMetadata()

Updated/new metadata files are saved to the module output directory (if created by the module). Use param = FALSE to return an empty file objects in preparation for building a new metadata file.

Returns:

Updated or new module/metadata/metadataFile.tsv is created, otherwise null

getModuleDir

File getModuleDir()

Each BioModule generates sub-directory under $DOCKER_PROJ

Returns:

BioModule root directory

getOutputDir

File getOutputDir()

Output files destined as input for the next BioModule is created in this directory.

Returns:

File directory containing the primary BioModule output

getPostRequisiteModules

List<String> getPostRequisiteModules()
                              throws Exception

Pipeline calls this method when building the list of pipeline BioModules to execute. Any BioModules returned by this method will be added to the pipeline after the current BioModule. If multiple post-requisites are found, the modules will be added in the order listed.

Returns:

List of BioModules

Throws:

Exception - if invalid Class names are returned as post-requisites

getPreRequisiteModules

List<String> getPreRequisiteModules()
                             throws Exception

Pipeline calls this method when building the list of pipeline BioModules to execute. Any BioModules returned by this method will be added to the pipeline before the current BioModule. If multiple prerequisites are returned, the modules will be added in the order listed.

Returns:

List of BioModule Class Names

Throws:

Exception - if invalid Class names are returned as prerequisites

getSummary

String getSummary()
           throws Exception

Gets the BioModule execution summary, this is sent as part of the notification email, if configured.
Summary should not include data content, to avoid unintentional publication of confidential information.
However, meta-data such as number/size of files can be helpful during debug.

Returns:

Summary of BioModule execution

Throws:

Exception - if any error occurs

getTempDir

File getTempDir()

Contains intermediate files generated by the module but not used by the next BioModule.
The files may contain supplementary information or data that may be helpful during debug or recovery.
If "pipeline.deleteTempFiles" = "Y", successful pipelines delete this directory.

Returns:

File directory of files typically not useful long term

getLogDir

File getLogDir()

Retains records of the process of running the module.
The files are intended to be small and stored long term with successful pipelines.

Returns:

getResourceDir

File getResourceDir()

The resource sub-direcotry contains files that the module may depend on.
Unlike temp or log, these files are not created by the module. Unlike output, these files are not intended to be available to other modules.
Examples of files that might go in resource/ include:
* library files that are used by a user-supplied script * files that are packaged in the jar file of a module, and copied to be available to scripts. * information that can be determined in the java layer, and is made available to non-java scripts.
Like all subdirs, this is deleted in incomplete modules during a re-start. Depending on the module, the files may be deleted upon module completion in the cleanup() method.

Returns:

init

void init()
   throws Exception

Initialize a new module to generate a unique ID and module directory.

Throws:

Exception - if errors occur

getPropDefault

String getPropDefault(String prop)

If a property is null based on the config files (including all defaults and standard.properties) but a module is passed to the Config class as the context for getting that module, the Config class can query the module for a value for the property. In general, this is used for the properties defined in this module, but it CAN also be used for general properties.

Parameters:

property -

Returns:

isValidInputModule

boolean isValidInputModule(BioModule previousModule)

BioModules getInputFiles() method typically, but not always, return the previousModule output files. This method checks the output directory from the previous module to check for input deemed acceptable by the current module. The conditions coded in this method will be checked on each previous module in the pipeline until acceptable input is found. If no previous module produced acceptable input, the files under Config."input.dirPaths" are returned.

This method can be overridden by modules that need input files generated prior to the previous module.

Parameters:

previousModule - BioModule that ran before the current BioModule

Returns:

boolean TRUE if the previousModule output is acceptable input for the current BioModule

getDockerImageOwner

String getDockerImageOwner()

Get the name of the docker hub user that owns the image to use for this module. Docker images from docker hub are specified using this syntax: /:

Returns:

getDockerImageName

String getDockerImageName()

Get the docker image to use for this module. The owner and version/tag are specificed separately; this is just the image name. Docker images from docker hub are specified using this syntax: /:

Returns:

getDockerImageTag

String getDockerImageTag()

Get the version / tag to use for the docker image. Docker images from docker hub are specified using this syntax: /:

Returns:

version

default String version()

Changes to a module class should be accompanied by a increment in version. The is a string and open form, but using the major.minor.patch semantic versioning is recommended.

Returns: