Autopsy  4.21.0
Graphical digital forensics platform for The Sleuth Kit and other tools.
Classes | Private Member Functions | Static Private Member Functions | Private Attributes | Static Private Attributes | List of all members
com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing Class Reference

Classes

class  FileRecord
 
class  IngestJobState
 

Private Member Functions

AnalysisResult createAnalysisResult (IngestJobState ingestJobState, SleuthkitCase.CaseDbTransaction trans, CTCloudBean cloudBean, Long objId) throws Blackboard.BlackboardException
 
void createAnalysisResults (IngestJobState ingestJobState, List< CTCloudBean > repResult, Map< String, List< Long >> md5ToObjId) throws Blackboard.BlackboardException, TskCoreException
 
List< CTCloudBeangetHashLookupResults (IngestJobState ingestJobState, List< String > md5Hashes) throws CTCloudException
 
IngestJobState getNewJobState (IngestJobContext context, boolean uploadFiles) throws Exception
 
void handleBatch (IngestJobState ingestJobState, List< FileRecord > fileRecords)
 
void handleLookupResults (IngestJobState ingestJobState, Map< String, List< Long >> md5ToObjId, List< CTCloudBean > repResult) throws Blackboard.BlackboardException, TskCoreException, TskCoreException, CTCloudException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
 
void handleNonFoundResults (IngestJobState ingestJobState, Map< String, List< Long >> md5ToObjId, List< CTCloudBean > results, boolean performFileUpload) throws CTCloudException, TskCoreException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
 
void longPollForNotFound (IngestJobState ingestJobState) throws InterruptedException, CTCloudException, Blackboard.BlackboardException, TskCoreException
 
boolean uploadFile (IngestJobState ingestJobState, String md5, long objId) throws CTCloudException, TskCoreException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
 

Static Private Member Functions

static String getOrCalcHash (AbstractFile af, HashType hashType)
 
static String getOrCalcMd5 (AbstractFile af)
 
static String getOrCalcSha1 (AbstractFile af) throws NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
 
static String getOrCalcSha256 (AbstractFile af)
 
static boolean isUploadable (AbstractFile af)
 
static String normalizedMd5 (String orig)
 
static void notifyWarning (String title, String message, Exception ex)
 
static long remaining (Long limit, Long used)
 

Private Attributes

final BatchProcessor< FileRecord > batchProcessor
 
final CTApiDAO ctApiDAO = CTApiDAO.getInstance()
 
final CTLicensePersistence ctSettingsPersistence = CTLicensePersistence.getInstance()
 
IngestJobState ingestJobState = null
 

Static Private Attributes

static final int BATCH_SIZE = 200
 
static final Set< String > EXECUTABLE_MIME_TYPES
 
static final long FILE_UPLOAD_RETRY_SLEEP_MILLIS = 60 * 1000
 
static final long FLUSH_SECS_TIMEOUT = 24 * 60 * 60
 
static final Logger logger = Logger.getLogger(MalwareScanIngestModule.class.getName())
 
static final long LOW_LOOKUPS_REMAINING = 250
 
static final long LOW_UPLOADS_REMAINING = 25
 
static final String MALWARE_CONFIG = ""
 
static final long MAX_UPLOAD_SIZE = 100_000_000
 
static final long MIN_UPLOAD_SIZE = 1
 
static final int NUM_FILE_UPLOAD_RETRIES = 7
 

Detailed Description

Does the bulk of processing for the ingest module and handles concurrent ingest modules adding files simultaneously.

Definition at line 103 of file MalwareScanIngestModule.java.

Member Function Documentation

AnalysisResult com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.createAnalysisResult ( IngestJobState  ingestJobState,
SleuthkitCase.CaseDbTransaction  trans,
CTCloudBean  cloudBean,
Long  objId 
) throws Blackboard.BlackboardException
private

Creates an analysis result for the given information.

Parameters
ingestJobStateThe state of the ingest job.
transThe case database transaction to use.
cloudBeanThe bean indicating the malware result.
objIdThe object id of the corresponding file that will receive the analysis result.
Returns
The created analysis result or null if none created.
Exceptions
org.sleuthkit.datamodel.Blackboard.BlackboardException

Definition at line 835 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.ctapi.json.MalwareResultBean.Status.FOUND, com.basistech.df.cybertriage.autopsy.ctapi.json.MalwareResultBean.getCTScore(), com.basistech.df.cybertriage.autopsy.ctapi.json.CTCloudBean.getMalwareResult(), com.basistech.df.cybertriage.autopsy.ctapi.json.MalwareResultBean.getStatus(), com.basistech.df.cybertriage.autopsy.ctapi.json.MalwareResultBean.getStatusDescription(), com.basistech.df.cybertriage.autopsy.ctapi.json.CTScore.getTskCore(), and com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.MALWARE_CONFIG.

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.createAnalysisResults().

void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.createAnalysisResults ( IngestJobState  ingestJobState,
List< CTCloudBean repResult,
Map< String, List< Long >>  md5ToObjId 
) throws Blackboard.BlackboardException, TskCoreException
private

Creates TSK_MALWARE analysis results based on a list of cloud beans received from the CT cloud api.

Parameters
ingestJobStateThe ingest job state.
repResultThe list of cloud beans. Only cloud beans with a malware status
md5ToObjIdThe mapping of md5
Exceptions
org.sleuthkit.datamodel.Blackboard.BlackboardException
TskCoreException

Definition at line 772 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.createAnalysisResult(), and com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.normalizedMd5().

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleLookupResults(), and com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.longPollForNotFound().

List<CTCloudBean> com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getHashLookupResults ( IngestJobState  ingestJobState,
List< String >  md5Hashes 
) throws CTCloudException
private
IngestJobState com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getNewJobState ( IngestJobContext  context,
boolean  uploadFiles 
) throws Exception
private
static String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcHash ( AbstractFile  af,
HashType  hashType 
)
staticprivate

Gets the md5 hash from the abstract file or calculates it.

Parameters
afThe abstract file.
Returns
The md5 hash (or null if could not be determined).

Definition at line 268 of file MalwareScanIngestModule.java.

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcMd5(), and com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcSha256().

static String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcMd5 ( AbstractFile  af)
staticprivate

Gets or calculates the md5 for a file.

Parameters
afThe file.
Returns
The hash.

Definition at line 308 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcHash().

static String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcSha1 ( AbstractFile  af) throws NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
staticprivate

Gets or calculates the sha1 for a file.

Parameters
afThe file.
Returns
The hash.

Definition at line 328 of file MalwareScanIngestModule.java.

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.uploadFile().

static String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcSha256 ( AbstractFile  af)
staticprivate
void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleBatch ( IngestJobState  ingestJobState,
List< FileRecord >  fileRecords 
)
private
void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleLookupResults ( IngestJobState  ingestJobState,
Map< String, List< Long >>  md5ToObjId,
List< CTCloudBean repResult 
) throws Blackboard.BlackboardException, TskCoreException, TskCoreException, CTCloudException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
private
void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleNonFoundResults ( IngestJobState  ingestJobState,
Map< String, List< Long >>  md5ToObjId,
List< CTCloudBean results,
boolean  performFileUpload 
) throws CTCloudException, TskCoreException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
private

Handles a CT cloud response objects that have a status that isn't FOUND but still are queryable (i.e. NOT_FOUND, BEING_SCANNED).

Parameters
ingestJobStateThe current state of operations of the ingest module.
md5ToObjIdThe mapping of md5 to a list of object ids.
resultsThe ct cloud results.
performFileUploadTrue if the class of results warrants file upload (i.e. NOT_FOUND)

Definition at line 509 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.normalizedMd5(), and com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.uploadFile().

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleLookupResults().

static boolean com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.isUploadable ( AbstractFile  af)
staticprivate

Whether or not an abstract file meets the requirements to be uploaded.

Parameters
afThe abstract file.
Returns
True if can be uploaded.

Definition at line 601 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.MAX_UPLOAD_SIZE.

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.uploadFile().

void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.longPollForNotFound ( IngestJobState  ingestJobState) throws InterruptedException, CTCloudException, Blackboard.BlackboardException, TskCoreException
private
static String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.normalizedMd5 ( String  orig)
staticprivate
static void com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.notifyWarning ( String  title,
String  message,
Exception  ex 
)
staticprivate
static long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.remaining ( Long  limit,
Long  used 
)
staticprivate
boolean com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.uploadFile ( IngestJobState  ingestJobState,
String  md5,
long  objId 
) throws CTCloudException, TskCoreException, NoSuchAlgorithmException, ReadContentInputStream.ReadContentInputStreamException
private

Uploads a file to CT Cloud if the file is valid for upload.

Parameters
ingestJobStateThe current state of the ingest job.
objIdThe object id of the file to upload to CT cloud.
Returns
True if successfully uploaded.
Exceptions
CTCloudException
TskCoreException

Definition at line 621 of file MalwareScanIngestModule.java.

References com.basistech.df.cybertriage.autopsy.ctapi.CTApiDAO.getAuthToken(), com.basistech.df.cybertriage.autopsy.ctapi.json.AuthTokenResponse.getFileUploadCount(), com.basistech.df.cybertriage.autopsy.ctapi.json.AuthTokenResponse.getFileUploadLimit(), com.basistech.df.cybertriage.autopsy.ctapi.json.AuthTokenResponse.getFileUploadUrl(), com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcSha1(), com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.getOrCalcSha256(), com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.isUploadable(), com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.notifyWarning(), com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.remaining(), com.basistech.df.cybertriage.autopsy.ctapi.json.MetadataUploadRequest.setCreatedDate(), com.basistech.df.cybertriage.autopsy.ctapi.json.FileUploadRequest.setFileName(), com.basistech.df.cybertriage.autopsy.ctapi.json.MetadataUploadRequest.setFileSizeBytes(), com.basistech.df.cybertriage.autopsy.ctapi.json.MetadataUploadRequest.setSha1(), com.basistech.df.cybertriage.autopsy.ctapi.json.MetadataUploadRequest.setSha256(), com.basistech.df.cybertriage.autopsy.ctapi.CTApiDAO.uploadFile(), and com.basistech.df.cybertriage.autopsy.ctapi.CTApiDAO.uploadMeta().

Referenced by com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.handleNonFoundResults().

Member Data Documentation

final int com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.BATCH_SIZE = 200
staticprivate

Definition at line 106 of file MalwareScanIngestModule.java.

final BatchProcessor<FileRecord> com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.batchProcessor
private
Initial value:
= new BatchProcessor<FileRecord>(
(lst) -> SharedProcessing.this.handleBatch(SharedProcessing.this.ingestJobState, lst))

Definition at line 141 of file MalwareScanIngestModule.java.

final CTApiDAO com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.ctApiDAO = CTApiDAO.getInstance()
private

Definition at line 147 of file MalwareScanIngestModule.java.

final CTLicensePersistence com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.ctSettingsPersistence = CTLicensePersistence.getInstance()
private

Definition at line 146 of file MalwareScanIngestModule.java.

final Set<String> com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.EXECUTABLE_MIME_TYPES
staticprivate
Initial value:
= Stream.of(
"application/x-bat",
"application/x-dosexec",
"application/vnd.microsoft.portable-executable",
"application/x-msdownload",
"application/exe",
"application/x-exe",
"application/dos-exe",
"vms/exe",
"application/x-winexe",
"application/msdos-windows",
"application/x-msdos-program"
).collect(Collectors.toSet())

Definition at line 123 of file MalwareScanIngestModule.java.

final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.FILE_UPLOAD_RETRY_SLEEP_MILLIS = 60 * 1000
staticprivate

Definition at line 121 of file MalwareScanIngestModule.java.

final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.FLUSH_SECS_TIMEOUT = 24 * 60 * 60
staticprivate

Definition at line 108 of file MalwareScanIngestModule.java.

IngestJobState com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.ingestJobState = null
private

Definition at line 149 of file MalwareScanIngestModule.java.

final Logger com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.logger = Logger.getLogger(MalwareScanIngestModule.class.getName())
staticprivate

Definition at line 139 of file MalwareScanIngestModule.java.

final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.LOW_LOOKUPS_REMAINING = 250
staticprivate

Definition at line 111 of file MalwareScanIngestModule.java.

final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.LOW_UPLOADS_REMAINING = 25
staticprivate

Definition at line 114 of file MalwareScanIngestModule.java.

final String com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.MALWARE_CONFIG = ""
staticprivate
final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.MAX_UPLOAD_SIZE = 100_000_000
staticprivate
final long com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.MIN_UPLOAD_SIZE = 1
staticprivate

Definition at line 117 of file MalwareScanIngestModule.java.

final int com.basistech.df.cybertriage.autopsy.malwarescan.MalwareScanIngestModule.SharedProcessing.NUM_FILE_UPLOAD_RETRIES = 7
staticprivate

The documentation for this class was generated from the following file:

Copyright © 2012-2022 Basis Technology. Generated on: Tue Feb 6 2024
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.