Brian Carrier
carrier at sleuthkit dot org
Issue #21
November 15, 2005
This issue of The Sleuth Kit Informer is long over due. Many asked if the Informer was no longer being published, but the truth is that there were too many other things going on. This issue is about changes that occured in the most recent releases of The Sleuth Kit and Autopsy. The first article is about new license changes and the second is about how the ils tool now handles FAT file systems. As a result of changes in the last version, ils may not show some unallocated directory entries in slack space, but the tool now runs MUCH faster.
I'm sure that I am missing some updates that have occurred since the last issue in May, but here are some of the major ones.
Versions 2.02 and 2.03 of The Sleuth Kit were released. The updates include support for Unicode in the FAT, NTFS, UFS, and ExtX file systems; improved handling of FAT short file names and deleted UFS and ExtX names; and more updates to the internal design of the code and their libraries. The Unicode update was supported by I.D.E.A.L. Technology. Autopsy 2.06 was released and it contains updates to support Unicode and now uses Cascading Style Sheets (CSS).
You can now buy Sleuth Kit-related merchandise, such as shirts and mugs, at the Sleuth Kit Cafe Press Store:
http://www.cafepress.com/sleuthkit/
There are now Italian translations of The Sleuth Kit Informer available from Maurizio Anconelli:
http://www.cybercrimes.it/
David Berger at NetMon has developed the Allin1 tool, which allows you to automatically perform some of the initial case setup in Autopsy in one step. It can:
The "Real Digital Forensics" book by Keith Jones, Richard Bejtlich, and Curtis Rose was recently released and it contains material on The Sleuth Kit and Autopsy:
http://www.realdigitalforensics.com/
The Sleuth Kit Informer is looking for articles on open source tools and techniques for digital investigations (computer / digital forensics) and incident response. Articles that discuss The Sleuth Kit and Autopsy are appreciated, but not required. Example topics include (but are not limited to):
http://www.sleuthkit.org/informer/cfp.html
By: Brian Carrier
The licenses for The Sleuth Kit changed with version 2.03. Prior versions had a combination of the IBM Public License (from the original TCT tools), the BSD license, and GNU Public License (GPL). This made it confusing for people who wanted to incorporate the analysis libraries into their own applications.
As of version 2.03, all TSK tools except srch_strings and file are released under the IBM Public License or the Common Public License (CPL). The CPL is the more general form of the IBM Public License. The srch_strings tool is from the GNU binutils package and the file tool has BSD license. Autopsy is still released under the GPL.
There were a couple of reasons for choosing the CPL. First, because it is equivalent to the IBM Public License, then all tools now have the same license. The architecture changes that have occurred over the past three years since the split from TCT would make it difficult to identify which code is released under which license. Second, the CPL allows the tools to be used as components in a closed source application while still requiring that updates made to TSK be open source.
While I would prefer that TSK be used only by other open source tools, I think there is an interesting compromise with forensic analysis tools and open and closed source. Three years ago, I wrote about the potential legal benefits from open source investigation tools in that they are a published procedure and allow an expert to testify about how the tools work. I also proposed an architecture that allows the data "extraction" tools to be open source and the "presentation" tools (or interface) to be closed source. This allows the investigator, or other expert, to verify her results and not put the vendor at risk of losing their trade secrets. After all, processing a FAT file system and other data formats that are well known does not require many trade secrets. This design allows companies to compete on interface, usability, support etc. and still have a design that is transparent and open to unbiased examination by the courts.
Licenses that require an open source tool to be used with only open source tools would have prevented the TSK libraries from being used in these situations. But, the CPL is one of those that supports the closed and open source combination. Therefore, it is now the license for many of the additions made since the split with TCT.
Brian Carrier. "Open Source Digital Forensic Tools: The Legal Argument." @stake Research Report. October 2002. Available at: http://www.digital-evidence.org/papers/opensrc_legal.pdf
Common Public License 1.0. Available at: http://www.opensource.org/licenses/cpl1.0.php
By: Brian Carrier
The ils tool in TSK lists the metadata information for allocated and unallocated files. With FAT file systems, this used to take a long time because ils was examining every sector of the file system. The 2.03 version of TSK changed how the FAT file system is processed and ils is now much faster, although the speed comes at a cost. The cost is that unallocated directory entries in the slack space of a file will not be listed. This article describes why this occurs.
The metadata for a FAT file includes the file size, cluster addresses, temporal data, and attributes. The data are stored in directory entry structures, which are located in the clusters allocated to the file's parent directory. This means that they can exist anywhere in the file system. Other file systems store all metadata in a central location, which makes them easier to find.
Because directory entries can be anywhere in the file system and TSK needs to find unallocated directory entries, the previous versions of TSK used to search every sector looking for directory entries. The original process was to examine the first 32 bytes of the sector (which is the size of a directory entry) and test them to see if they could be a valid directory entry. If they were, then the entire sector would be processed and if they were not then the next sector was examined. This is why it took so long.
Version 2.02 changed this process slightly because some file systems were found where the first entry in the sector had a deleted entry and it did not pass the tests. Therefore, the full sector was skipped even though it contained allocated entries. This meant that files could be found by fls when it was processing the directory contents, but the files would not be found using ils. The fix was to first determine which sectors were allocated to directories and never skip those, even if the first 32 bytes was not a valid directory entry. This unfortunately made the process even longer because it had to recurse the directory hierarchy before examining every sector.
Version 2.03 added support for Unicode, which means that the test on the first 32 bytes had to be relaxed to support file names using code pages instead of only ASCII. This change resulted in many false positives caused by random data that were not actually directory entries. Therefore, the entire approach had to be reconsidered. The result is a much more efficient design that examines only sectors that are either allocated to directories or unallocated. In other words, it skips the sectors allocated to files (which should not have directory entries in them anyway).
This speeds up the process dramatically and reduces the false positives, but the downside is that there could be directory entries in the slack space of files. The final sectors allocated to a file (the slack space) could contain data from the file or directory that previously allocated it. If a directory previously allocated the sector then the sector could contain directory entries, but ils no longer examines or displays them.
It is not clear how common directory entries in slack spare are and it is not yet clear which approach other tools take. This is an example of how not having standard analysis procedures can produce different results with different tools. In the future, I will examine how expensive it is to include slack space into the new design.
Brian Carrier. "File System Forensic Analysis." Addison Wesley. 2005.