biolockj.module.implicit

Class Demultiplexer

java.lang.Object

biolockj.module.BioModuleImpl

biolockj.module.ScriptModuleImpl

biolockj.module.JavaModuleImpl

biolockj.module.implicit.Demultiplexer

All Implemented Interfaces:

ApiModule, BioModule, JavaModule, ScriptModule, SeqModule, Comparable<BioModule>

public class Demultiplexer
extends JavaModuleImpl
implements SeqModule, ApiModule

This BioModule splits multiplexed data into a separate file or pair of files (for paired reads) for each sample. The sample ID, or an identification barcode must be included in the header of each read. Barcodes are matched to Sample IDs using the metadata column configured via property "metadata.barcodeColumn".
If demultiplexer.strategy=barcode_in_header, register a match if the barcode is found anywhere in the sequence header.
If demultiplexer.strategy=barcode_in_seq, register a match if the sequence itself starts with any of the barcodes. The output multiplexed file will have barcodes removed from the sequences since they no longer serve any purpose.
Sequences with no matching barcode, or without a matching paired header (if paired), are output to a "NO_MATCH" file in the module temp directory.
Paired reads are accepted in 2 multiplexed formats:

1. Forward and reverse reads can be partitioned into 2 separate files identified by identified by: input.suffixFw and input.suffixRv.
2. Forward and reverse reads can be combined in a single file with read direction indicated in the header by values "1:N:" and "2:N:"

GUI Module Name

Demultiplexer

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	NUM_LINES_TEMP_FILE Module splits multiplexed file into smaller files with this number of lines: 2000000

Fields inherited from class biolockj.module.JavaModuleImpl

BLJ_OPTIONS

Fields inherited from class biolockj.module.BioModuleImpl

GZIP_EXT, LOG_EXT, PDF_EXT, RETURN, SH_EXT, TAB_DELIM, TSV_EXT, TXT_EXT

Fields inherited from interface biolockj.module.BioModule

LOG_DIR, MAIN_SCRIPT_PREFIX, NO_VERSION, OUTPUT_DIR, RES_DIR, TEMP_DIR

Constructor Summary

Constructors

Constructor and Description
Demultiplexer()

Method Summary

Modifier and Type	Method and Description
protected void	breakUpFiles() Some multiplexed files can be very large.
void	checkDependencies() Validate module dependencies: If Config."demultiplexer.strategy" indicates use of barcodes to demultiplexer, validate metadata column named Config."metadata.barcodeColumn" exists Call DemuxUtil.setMultiplexedConfig() to set multiplexed Config if needed If Config."demultiplexer.barcodeCutoff" defined, validate between 0.0 - 1.0
void	cleanUp() Update SeqUtil to indicate data has been demultiplexed.
protected void	demultiplex(Map<String,Set<String>> validHeaders) Demultiplex the file into separate small temp files, with 2000000 lines each for processing.
String	getCitationString() At a minimum, this should return the name and/or url for the wrapped tool.
String	getDescription() Briefly describe what this module does.
String	getDetails() A extension of getDescription.
List<File>	getInputFiles() BioModule BioModuleImpl.getInputFiles() is called to initialize upon first call and cached.
List<File>	getSeqFiles(Collection<File> files) Return only sequence files for sample IDs found in the metadata file. If Config."metadata.required" = "Y", an error is thrown to list the files that cannot be matched to a metadata row.
String	getSummary() Produces initial count and demultiplexed output count summaries for forward/reverse reads.
protected Map<String,Set<String>>	getValidFwHeaders() Get valid forward read headers that belong to reads with a valid barcode or sample identifier.
protected Map<String,Set<String>>	getValidHeaders() This method obtains all valid headers for the forward reads, and returns only headers that also have a matching reverse read
void	runModule() Module execution summary: Execute breakUpFiles() to split the multiplex file into smaller size for processing Execute getValidHeaders() to obtain list of valid headers matched to sample ID with the metadata file and also verifies matching forward and reverse read headers if demuliplexing paired reads.

Methods inherited from class biolockj.module.JavaModuleImpl

buildScript, executeTask, getDockerImageName, getDockerImageOwner, getDockerImageTag, getWorkerScriptFunctions, isValidInputModule, markStatus, moduleComplete, moduleFailed, runBioLockJ_CMD

Methods inherited from class biolockj.module.ScriptModuleImpl

buildScriptForPairedReads, getJobParams, getMainScript, getRuntimeParams, getScriptDir, getScriptErrors, getTimeout, hasScripts, isValidProp

Methods inherited from class biolockj.module.BioModuleImpl

addGeneralProperty, addGeneralProperty, addGeneralProperty, addNewProperty, addNewProperty, cacheInputFiles, compareTo, equals, findModuleInputFiles, getAlias, getDescription, getFileCache, getID, getLogDir, getMenuPlacement, getMetadata, getModuleDir, getOutputDir, getPostRequisiteModules, getPreRequisiteModules, getPropDefault, getPropDescMap, getPropType, getPropTypeMap, getResourceDir, getTempDir, getTitle, hashCode, init, listProps, setAlias, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface biolockj.module.ScriptModule

buildScript, buildScriptForPairedReads, getJobParams, getMainScript, getScriptDir, getScriptErrors, getTimeout, getWorkerScriptFunctions

Methods inherited from interface biolockj.module.BioModule

executeTask, getAlias, getDockerImageName, getDockerImageOwner, getDockerImageTag, getID, getLogDir, getMetadata, getModuleDir, getOutputDir, getPostRequisiteModules, getPreRequisiteModules, getPropDefault, getResourceDir, getTempDir, init, isValidInputModule, setAlias, version

Methods inherited from interface biolockj.api.ApiModule

getDescription, getMenuPlacement, getPropType, getTitle, isValidProp, listProps

Field Detail

NUM_LINES_TEMP_FILE

protected static final int NUM_LINES_TEMP_FILE

Module splits multiplexed file into smaller files with this number of lines: 2000000

See Also:

Constant Field Values

Constructor Detail

Demultiplexer

public Demultiplexer()
              throws API_Exception

Throws:

API_Exception

Method Detail

checkDependencies

public void checkDependencies()
                       throws Exception

Validate module dependencies:

1. If Config."demultiplexer.strategy" indicates use of barcodes to demultiplexer, validate metadata column named Config."metadata.barcodeColumn" exists
2. Call DemuxUtil.setMultiplexedConfig() to set multiplexed Config if needed
3. If Config."demultiplexer.barcodeCutoff" defined, validate between 0.0 - 1.0

Specified by:

checkDependencies in interface BioModule

Overrides:

checkDependencies in class ScriptModuleImpl

Throws:

Exception - thrown if missing or invalid dependencies are found

cleanUp

public void cleanUp()
             throws Exception

Update SeqUtil to indicate data has been demultiplexed.

Specified by:

cleanUp in interface BioModule

Overrides:

cleanUp in class BioModuleImpl

Throws:

Exception - if unable to modify property

getInputFiles

public List<File> getInputFiles()

Description copied from class: BioModuleImpl

BioModule BioModuleImpl.getInputFiles() is called to initialize upon first call and cached.

Specified by:

getInputFiles in interface BioModule

Overrides:

getInputFiles in class BioModuleImpl

Returns:

Input files

getSeqFiles

public List<File> getSeqFiles(Collection<File> files)
                       throws SequnceFormatException

Description copied from interface: SeqModule

Return only sequence files for sample IDs found in the metadata file.
If Config."metadata.required" = "Y", an error is thrown to list the files that cannot be matched to a metadata row.

Specified by:

getSeqFiles in interface SeqModule

Parameters:

files - Module input files

Returns:

Module sequence files

Throws:

SequnceFormatException - If Config."metadata.required" = "Y" but sequence files found that do not have a corresponding record in the metadata file or if invalid metadata prevents parsing SEQ files.

getSummary

public String getSummary()
                  throws Exception

Produces initial count and demultiplexed output count summaries for forward/reverse reads.

Specified by:

getSummary in interface BioModule

Overrides:

getSummary in class ScriptModuleImpl

Returns:

Summary of BioModule execution

Throws:

Exception - if any error occurs

runModule

public void runModule()
               throws Exception

Module execution summary:

1. Execute breakUpFiles() to split the multiplex file into smaller size for processing
2. Execute getValidHeaders() to obtain list of valid headers matched to sample ID with the metadata file and also verifies matching forward and reverse read headers if demuliplexing paired reads.
3. Execute demultiplex(Map) to demultiplex the data into a separate file (or pair of files) for each sample

If paired reads are combined in a single file the read direction must be identified in the sequence header using key strings " 1:N:" " 2:N:"

Specified by:

runModule in interface JavaModule

Specified by:

runModule in class JavaModuleImpl

Throws:

Exception - thrown if any runtime error occurs

breakUpFiles

protected void breakUpFiles()
                     throws Exception

Some multiplexed files can be very large. This method breaks each input file into smaller files, each 2000000 lines in size to avoid memory issues while processing.

Throws:

Exception - if unexpected errors occur at runtime

demultiplex

protected void demultiplex(Map<String,Set<String>> validHeaders)
                    throws Exception

Demultiplex the file into separate small temp files, with 2000000 lines each for processing. If paired reads, generate separate forward and reverse files for each sample output file named by sample ID.

Parameters:

validHeaders - Set of valid headers

Throws:

Exception - if error occurs reading the multiplexed file

getValidFwHeaders

protected Map<String,Set<String>> getValidFwHeaders()
                                             throws Exception

Get valid forward read headers that belong to reads with a valid barcode or sample identifier.

Returns:

Map sampleID to list of valid headers

Throws:

Exception - if error occur

getValidHeaders

protected Map<String,Set<String>> getValidHeaders()
                                           throws Exception

This method obtains all valid headers for the forward reads, and returns only headers that also have a matching reverse read

Returns:

Map of valid headers for each sample ID

Throws:

Exception - if unable to obtain headers

getDescription

public String getDescription()

Description copied from interface: ApiModule

Briefly describe what this module does. A more detailed description may be returned by getDetails.

Specified by:

getDescription in interface ApiModule

Returns:

getDetails

public String getDetails()
                  throws API_Exception

Description copied from interface: ApiModule

A extension of getDescription. Beyond the brief description, give details such as the interaction between properties.

Specified by:

getDetails in interface ApiModule

Overrides:

getDetails in class BioModuleImpl

Returns:

Throws:

API_Exception

getCitationString

public String getCitationString()

Description copied from interface: ApiModule

At a minimum, this should return the name and/or url for the wrapped tool. For BioLockJ home-grown modules, it can cite BioLockJ and give the current version. Ideally this will include the version for the wrapped tool. Tool version may require using a stored variable that can be filled in during execution; and the string that is returned pre-execution may not have the version.

Specified by:

getCitationString in interface ApiModule

Returns: