Autopsy  4.1
Graphical digital forensics platform for The Sleuth Kit and other tools.
Developing Ingest Modules

Getting Started

This page describes how to develop ingest modules using either Java or Python (Jython). It assumes you have already set up your development environment as described in Java Development Setup or Python Development Setup.

Ingest Module Types

Ingest modules analyze data from a data source (e.g., a disk image or a folder of logical files). There are two types of ingest modules in Autopsy:

The difference between these two types of modules is what gets passed in to them:

Here are some guidelines for choosing the type of your ingest module:

As you will learn a little later in this guide, it is possible to make an ingest module that has both file-level and data-source level capabilities. You would do this when you need to work at both levels to get all of your analysis done. The modules in such a pair will be enabled or disabled together and will have common per ingest job and global settings.

The text below will refer to example code in the org.sleuthkit.autopsy.examples package. The sample modules don't do anything particularly useful, but they can serve as templates for developing your own ingest modules.

Ingest Module Life Cycle

Before we dive into the details of creating a module, it is important to understand the life cycle of the module. Note that this life cycle is much different for Autopsy 3.1 and newer modules compared to Autopsy 3.0 modules. This section only talks about 3.1 and newer modules.

You will need to implement at least two classes to make an ingest module:

  1. A factory class that will be created when Autopsy starts and will provide configuration panels to Autopsy and will create instances of your ingest module.
  2. An ingest module class that will be instantiated by the factory when the ingest modules are run. A new instance of this class will be created for each ingest thread.

Here is an example sequence of events. Details will be provided below.

  1. User launches Autopsy and it looks for classes that implement the org.sleuthkit.autopsy.ingest.IngestModuleFactory interface.
  2. Autopsy finds and creates an instance of your FooIngestModuleFactory class.
  3. User adds a disk image.
  4. Autopsy presents the list of available ingest modules to the user and uses the utility methods from FooIngestModuleFactory class to get the module's name, description, and configuration panels.
  5. User enables your module (and others).
  6. Autopsy uses FooIngestModuleFactory to create two instances of FooIngestModule (Autopsy is using two threads to process the files).
  7. Autopsy calls FooIngestModule.startUp() on each thread and then calls FooIngestModule.process() to pass in each file.

Creating a Basic Ingest Module

Basic Ingest Module Factory

The first step to write an ingest module is to make its factory. There are three general types of things that a factory does:

  1. Provides basic information such as the module's name, version, and description. (required)
  2. Creates ingest modules. (required)
  3. Provides panels so that the user can configure the module. (optional)

This section covers the required parts of a basic factory so that we can make the ingest module. A later section (User Options and Configuration) covers how you can use the factory to provide options to the user.

To make writing a simple factory easier, Autopsy provides an adapter class that implements the "optional" methods in the interface. Our basic factory will use the adapter.

  1. Make a factory class by either:
  2. Update and create the needed methods using the org.sleuthkit.autopsy.ingest.IngestModuleFactory interface documentation. You can also use the sample code mentioned above for examples of what the methods should do if you did not already copy and paste them.
  3. If you are using Java, import org.openide.util.lookup.ServiceProvider and add a dependency on the NetBeans Lookup API module to the NetBeans module that contains your ingest module. Then add a NetBeans ServiceProvider annotation so that the factory is found at run time:
    @ServiceProvider(service = IngestModuleFactory.class)

At this point, when you add a data source to an Autopsy case, you should see the module in the list of ingest modules. If you don't see it, double check that you either implemented org.sleuthkit.autopsy.ingest.IngestModuleFactory or extended or inherited org.sleuthkit.autopsy.ingest.IngestModuleFactoryAdapter. If using Java, make sure that you added the service provider annotation.

Common Concepts of Ingest Modules

Before we cover the specific interfaces of the two different types of modules, let's talk about some common things.

Creating a Data Source-level Ingest Module

To create a data source ingest module:

  1. Create the ingest module class by either:
  2. Configure your factory class to create instances of the new ingest module class. To do this, you will need to change the isDataSourceIngestModuleFactory() method to return true and have the createDataSourceIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used. Your factory should have code similar to this Java code:
@Override
public boolean isDataSourceIngestModuleFactory() {
return true;
}
@Override
public DataSourceIngestModule createDataSourceIngestModule(IngestModuleIngestJobSettings ingestOptions) {
return new FooDataSourceIngestModule(); // replace this class name with the name of your class
}
  1. Use this page, the sample, and the documentation for the org.sleuthkit.autopsy.ingest.DataSourceIngestModule interfaces to implement the startUp() and process() methods.

Note that data source ingest modules must find the files that they want to analyze. The best way to do that is using one of the findFiles() methods of the org.sleuthkit.autopsy.casemodule.services.FileManager class. See Framework Services and Utilities for more details.

Creating a File Ingest Module

To create a file ingest module: To create a data source ingest module:

  1. Create the ingest module class by either:
  2. Configure your factory class to create instances of the new ingest module class. To do this, you will need to change the isFileIngestModuleFactory() method to return true and have the createFileIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used. Your factory should have code similar to this Java code:
@Override
public boolean isFileIngestModuleFactory() {
return true;
}
@Override
public FileIngestModule createFileIngestModule(IngestModuleIngestJobSettings ingestOptions) {
return new FooFileIngestModule(); // replace this class name with the name of your class
}
  1. Use this page, the sample, and the documentation for the org.sleuthkit.autopsy.ingest.FileIngestModule interface to implement the startUp(), and process(), and shutDown() methods.
    • org.sleuthkit.autopsy.ingest.FileIngestModule.startUp() should have any code that you need to initialize your module. If you have any startup errors, be sure to throw a IngestModuleException exception to stop ingest.
    • org.sleuthkit.autopsy.ingest.FileIngestModule.process() is where all of the work of a file ingest module is done. It will be called repeatedly between startUp() and shutDown(), once for each file Autopsy feeds into the pipeline of which the module instance is a part. The process() method receives a reference to a org.sleuthkit.datamodel.AbstractFile object.

Next Steps

This section gave you the outline of making the module. Now we'll cover what you can do in the module. The following sections often make use of the org.sleuthkit.autopsy.ingest.IngestServices class, which provides many convenient services to make module writing easier. Also make sure you refer to Framework Services and Utilities if you are looking for a feature.

Doing Something With Your Results

The previous section outlined how to make the basic module and how to get access to the data. The next step is to then do some fancy analysis and present the results to the user.

The first question that you must answer is what type of data do you want the user to see. There are two options:

  1. Data that can be accessed from the tree on the left-hand side of the UI and can be displayed in a table. To do this, you will make blackboard artifacts.
  2. Data that is in a big text file or some other report that the user can review. To do this, you will use the Case.addReport() method to make the output available in the directory tree.

Posting Results to the Blackboard

The blackboard is used to store results so that they are displayed in the results tree. See The Blackboard for details on posting results to it. You use the blackboard when you have specific items to show the user. if you want to just shown them a big report from another library or tool, see Developing Report Modules. The blackboard defines artifacts for specific data types (such as web bookmarks). You can use one of the standard artifact types or create your own.

When modules add data to the blackboard, they should notify listeners of the new data by invoking the org.sleuthkit.autopsy.ingest.IngestServices.fireModuleDataEvent() method. Do so as soon as you have added an artifact to the blackboard. This allows other modules (and the main UI) to know when to query the blackboard for the latest data. However, if you are writing a large number of blackboard artifacts in a loop, it is better to invoke org.sleuthkit.autopsy.ingest.IngestServices.fireModuleDataEvent() only once after the bulk write, so as not to flood the system with events.

Further, when modules create artifacts, they should be indexed for keyword search, using the method org.sleuthkit.autopsy.casemodule.services.Blackboard.indexArtifact(BlackboardArtifact artifact). This can be done in the following way:

Blackboard blackboard = Case.getCurrentCase().getServices().getBlackboard();
try {
blackboard.indexArtifact(artifact); //Your artifact as the argument.
}
catch (BlackboardException ex) {
//YOUR EXCEPTION BEHAVIOR HERE.
}

Making a Report

If your module makes a text or HTML file (or some other report format) that has some complex structure (either because you prefer to write output that way or your module is simply a wrapper around another tool), then you can simply call the output a report and then it will be shown in the UI in the reports area. You can do this by calling the org.sleuthkit.autopsy.casemodule.Case.addReport() method.

Getting User Attention

The ingest modules are running in the background and the user will not notice everything you put in the tree.

Posting Results to the Message Inbox

Modules should post messages to the inbox when interesting data is found. Of course, such data should also be posted to the blackboard as described above. The idea behind the ingest messages is that they are presented in chronological order so that users can see what was found while they were focusing on something else.

Inbox messages should only be sent if the result has a low false positive rate and will likely be relevant. For example, the core Autopsy hash lookup module sends messages if known bad (notable) files are found, but not if known good (NSRL) files are found. This module also provides a global setting (using its global settings panel) that allows a user to turn these messages on or off.

Messages are created using the org.sleuthkit.autopsy.ingest.IngestMessage class and posted to the inbox using the org.sleuthkit.autopsy.ingest.IngestServices.postMessage() method.

Reporting Errors

When an error occurs, you should write an error message to the Autopsy logs, using a logger obtained from org.sleuthkit.autopsy.ingest.IngestServices.getLogger().

You could also send an error message to the ingest inbox. The downside of this is that the ingest inbox was not really designed for this purpose and it is easy for the user to miss these messages. Therefore, it is preferable to post a pop-up message that is displayed in the lower right hand corner of the main window by calling org.sleuthkit.autopsy.coreutils.MessageNotifyUtil.Notify.show().

User Options and Configuration

Autopsy allows a module to provide two levels of configuration:

To provide either or both of these options to the user, we need to implement methods defined in the IngestModuleFactory interface. You can either add them to your class that extends the IngestModuleFactoryAdapter or decide to simply implement the interface.

You can also refer to sample implementations of the interfaces and abstract classes in the org.sleuthkit.autopsy.examples package, although you should note that the samples do not do anything particularly useful.

Ingest Job Options

Autopsy allows you to provide a graphical panel that will be displayed when the user decides to enable the ingest module. This panel is supposed to be for settings that the user may turn on or off for different data sources.

To provide options for each ingest job:

Your panel should create the IngestModuleIngestJobSettings class to store the settings and that will be passed back into your factory with each call to createDataSourceIngestModule() or createFileIngestModule(). The way that we have implemented this in Autopsy modules is that the factory casts the IngestModuleINgestJobSettings object to the module-specific implementation and then passes it into the constructor of the ingest module. The ingest module can then call whatever getter methods that were defined based on the panel settings.

You can also implement the getDefaultIngestJobSettings() method to return an instance of your IngestModuleIngestJobSettings class with default settings. Autopsy will call this when the module has not been run before.

NOTE: We recommend storing simple data in the IngestModuleIngestJobSettings-based class. In the case of our hash lookup module, we store the string names of the hash databases to do lookups in. We then get the hash database handles in the call to startUp() using the global module settings.

NOTE: The main benefit of using the IngestModuleIngestJobSettings-based class to store settings (versus some static variables in your package) are:

Global Options

Global options are those that are not specific to a data source or ingest pipeline. They are big-picture settings.

To provide global options:

Migrating 3.0 Java Ingest Modules to the 3.1 and newer API

This section is a guide for module developers who wrote modules for the 3.0 API. These API changes occurred so that we could make parallel pipelines of the file-level ingest modules. This section assumes you've read the above description of the new API.

There are three big changes to make in your module:

  1. Modules are no longer singletons. Autopsy will make one of your factory classes and many instances of the ingest modules. As part of the migration to the new classes, your singleton infrastructure will disappear.
  2. You'll need to move the UI/Configuration methods into the factory class and the ingest module methods into their own class. You'll also need to update the APIs for the methods a bit.
  3. You'll need to review your ingest module code for thread safety if you are using any static member variables.

We recommend that you:

  1. Create a new factory class and move over the UI panels, configuration code, and standard methods (name, description, version, etc.). You'll probably want the name in the ingest module code, so you should also store the name in a package-wide static member variable.
  2. Get the factory to compile and work. You can do basic testing by running Autopsy and verifying that you see your module and its panels.
  3. Change your old ingest module to implement the new interface and adjust it (see the name changes below). Then update the factory to create it.
  4. Review the ingest module code for thread safety (especially look for static member variables).

The following table provides a mapping of the methods of the old abstract classes to the new interfaces:

Old method New Method
IngestModuleAbstract.getType() N/A
IngestModuleAbstract.init() IngestModule.startUp()
IngestModuleAbstract.getName() IngestModuleFactory.getModuleName()
IngestModuleAbstract.getDescription() IngestModuleFactory.getModuleDescription()
IngestModuleAbstract.getVersion() IngestModuleFactory.getModuleVersion()
IngestModuleAbstract.hasBackgroundJobsRunning N/A
IngestModuleAbstract.complete() IngestModule.shutDown() for file ingest modules; data source ingest modules should do anything they did in complete() at the end of the process() method
IngestModuleAbstract.hasAdvancedConfiguration() IngestModuleFactory.hasGlobalSettingsPanel()
IngestModuleAbstract.getAdvancedConfiguration() IngestModuleFactory.getGlobalSettingsPanel()
IngestModuleAbstract.saveAdvancedConfiguration() IngestModuleGlobalSetttingsPanel.saveSettings()
N/A IngestModuleFactory.getDefaultIngestJobSettings()
IngestModuleAbstract.hasSimpleConfiguration() IngestModuleFactory.hasIngestJobSettingsPanel()
IngestModuleAbstract.getSimpleConfiguration() IngestModuleFactory.getIngestJobSettingsPanel()
IngestModuleAbstract.saveSimpleConfiguration() N/A
N/A IngestModuleIngestJobSettingsPanel.getSettings()
N/A IngestModuleFactory.isDataSourceIngestModuleFactory()
N/A IngestModuleFactory.createDataSourceIngestModule()
N/A IngestModuleFactory.isFileIngestModuleFactory()
N/A IngestModuleFactory.createFileIngestModule()

Notes:

Copyright © 2012-2016 Basis Technology. Generated on: Tue Oct 25 2016
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.