MD5DEEP(1)                  United States Air Force                 MD5DEEP(1)



NAME
       md5deep - Compute and compare MD5 message digests
       sha1deep - Compute and compare SHA-1 message digests
       sha256deep - Compute and compare SHA-256 message digests
       sha3deep - Compute and compare SHA-3-256 message digests
       tigerdeep - Compute and compare Tiger message digests
       whirlpooldeep - Compute and compare Whirlpool message digests


SYNOPSIS
       md5deep -v | -V | -h
       md5deep  [-m|-M|-x|-X  <file>]   [-a|-A <hash>] [-f <file>] [-p <size>]
       [-i <size>] [-tnwzresS0lbkqZud] [-F <bum>] [-o <fbcplsde>]  [-j  <num>]
       [[FILES]


DESCRIPTION
       Computes  the  hashes, or message digest, for any number of files while
       optionally recursively digging through the  directory  structure.   Can
       also  take  a  list  of known hashes and display the filenames of input
       files whose hashes either do or do not match any of the  known  hashes.
       Errors are reported to standard error. If no FILES are specified, reads
       from standard input.


       -p <size>
              Piecewise mode. Breaks files into chunks before hashing.  Chunks
              may  be  specified using IEC multipliers b, k, m, g, t, p, or e.
              (Never let it be said that the author didn't plan ahead!)   This
              mode cannot be used with the -z mode.


       -i|-I <size>
              Size  threshold mode. Only hash files smaller than the given the
              threshold. In -i mode, simply omits those files larger than  the
              threshold.  In  -I  mode, displays all files, but uses asterisks
              for the hashes of files larger than the threshold.  Sizes may be
              specified using IEC multipliers b, k, m, g, t, p, or e.


       -r     Enables recursive mode. All subdirectories are traversed. Please
              note that recursive mode cannot be used to examine all files  of
              a  given  file  extension. For example, calling md5deep -r *.txt
              will examine all files in directories that end in .txt.


       -e     Displays a progress indicator and estimate of time remaining for
              each  file being processed. Time estimates for files larger than
              4GB are not available on Windows. This mode may not be used with
              th -p mode.


       -m <file>
              Enables  matching mode. The file given should be a list of known
              hashes.  The input files are examined one at a  time,  and  only
              those files that match the list of known hashes are output. This
              flag may be used more than once to add multiple  sets  of  known
              hashes.  Acceptable  formats for lists of known hashes are plain
              (such as those  generated  by  md5deep  or  md5sum),  Hashkeeper
              files, iLook, and the National Software Reference Library (NSRL)
              as produced by the National Institute for Standards in  Technol‐
              ogy.
               If standard input is used with the -m flag, displays "stdin" if
              the input matches one of the hashes in the list of known hashes.
              If the hash does not match, the program displays no output.
               This flag may not be used in conjunction with the -x, -X, or -A
              flags.  See the section "UNICODE SUPPORT" below.


       -x <file>
              Same as the -m flag above, but does negative matching. That  is,
              only those files NOT in the list of known hashes are displayed.
               This flag may not be used in conjunction with the -m, -M, or -a
              flags.  See the section "UNICODE SUPPORT" below.

       -M and -X <file>
              Same as -m and -x above, but displays the  hash  for  each  file
              that does (or does not) match the list of known hashes.


       -a <hash>
              Adds a single hash to the list of known hashes used for matching
              mode, and if not already enabled, enables matching mode.  Adding
              single  hashes cannot, by itself, be used to print the hashes of
              matching files like the -M flag does. When used  in  conjunction
              with  the  -w flag, the filename displayed is just the hash sub‐
              mitted on the command line.
               This flag may not be used in conjunction with the -x, -X, or -A
              flags.


       -A <hash>
              Same as -a above, but does negative matching.  This flag may not
              be used in conjunction with the -m, -M, or -A flags.


       -f <file>
              Takes a list of files to be hashed from the specified file. Each
              line  is  assumed  to  be a filename. This flag can only be used
              once per invocation. If it's used  a  second  time,  the  second
              instance will clobber the first.
              Note  that  you  can still use other flags, such as the -m or -x
              modes, and submit additional FILES on the command line.


       -w     During any of the matching modes (-m,-M,-x,or -X), displays  the
              filename of the known hash that matched the input file.  See the
              section "UNICODE SUPPORT" below.


       -t     Display a timestamp in GMT with each  result.  On  Windows  this
              timestamp will be the file's creation time. On all other systems
              it should be the file's change time.


       -n     During any of the matching modes (-m,-M,-x,or -X), displays only
              the  filenames  of any known hashes that were not matched by any
              of the input files.


       -s     Enables silent mode. All error messages are supressed.


       -S     Like silent mode, but still displays warnings on improperly for‐
              matted hashes in the list of known hashes.


       -z     Enables  file size mode. Prepends the hash with a ten digit rep‐
              resentation of the size of each file processed. If the file size
              is  greater than 9999999999 bytes (about 9.3GB) the program dis‐
              plays 9999999999 for the size.


       -q     Quiet mode. File names are omitted from the output. Each hash is
              still followed by two spaces before the newline.


       -Z     Produces  output  in Triage format. Each line contans the file's
              size, a tab, a hash of the first 512 bytes, a tab, the  hash  of
              the  complete  file, a tab, and the file name.  These values are
              intended in increasing order of specificity. That is, two  files
              with  different sizes cannot possibly match. This is a fast com‐
              parison and should be done first. Next, two files with different
              partial  hashes cannot possibly match. This is often faster than
              hashing the whole file. Finally, if those two pieces align, then
              it's worth reading and hashing the entire file.


       -0     Uses  a  NULL character (/0) to terminate each line instead of a
              newline.  Useful for processing filenames with  strange  charac‐
              ters.


       -l     Enables  relative  file  paths. Instead of printing the absolute
              path for each file, displays the relative file path as indicated
              on  the  command  line. This flag may not be used in conjunction
              with the -b flag.


       -b     Enables bare mode. Strips any leading directory information from
              displayed  filenames.   This flag may not be used in conjunction
              with the -l flag.


       -k     Enables asterisk mode. An asterisk is inserted in lieu of a sec‐
              ond space between the filename and the hash, just like md5sum in
              its binary (-b) mode.


       -c     Enables comma separated values output, or CSV  mode.  This  mode
              has  the  side  effect  of removing the 10 digit size limitation
              from -z mode.  Also note that asterisks from  -k  mode  are  not
              displayed when in CSV mode.


       -o <bcpflsd>
              Enables  expert  mode.  Allows  the user specify which (and only
              which) types of files are  processed.  Directory  processing  is
              still  controlled  with  the  -r  flag.  The expert mode options
              allowed are:
              f - Regular files
              b - Block Devices
              c - Character Devices
              p - Named Pipes
              l - Symbolic Links
              s - Sockets
              d - Solaris Doors
              e - Windows PE executables


       -jnn   Controls multi-threading. By default the program will create one
              producer  thread  to scan the file system and one hashing thread
              per CPU core. Multi-threading causes output filenames to  be  in
              non-deterministic  order, as files that take longer to hash will
              be delayed while they are hashed. If a  deterministic  order  is
              required, specify -j0 to disable multi-threading


       -d     Output in Digital Forensics XML (DFXML) format.


       -u     Quote  Unicode  output.  For  example,  the  snowman is shown as
              U+C426.


       -F<bum>
              Specifies the input mode that is used to read files. The default
              is -Fb (buffered I/O) which reads files with fopen(). Specifying
              -Fu will use unbuffered I/O and read the file with open(). Spec‐
              ifying  -Fm  will  use memory-mapped I/O which will be faster on
              some platforms, but which (currently) will not work  with  files
              that produce I/O errors.


       -h     Show a help screen and exit.


       -v     Show the version number and exit.


       -V     Show copyright information and exit.


UNICODE SUPPORT
       As  of version 3.0 the program supports Unicode characters in filenames
       on Microsoft Windows systems for filenames  specified  on  the  command
       line  with  globbing (e.g. *), for files specified with the -f of files
       to hash, and for files read from directories using the -r option.

       By default all program input and output should be in UTF-8.   The  pro‐
       gram automatically converts this to UTF-16 for opening files).

       On  Unix/Linux/MacOS,  you should use a terminal emulator that supports
       UTF-8 and UTF-8 characters in filenames will be properly displayed.

       On Windows, the programs do not display Unicode characters on the  con‐
       sole.  You must either redirect output to a file and open the file with
       Wordpad (which can display Unicode), or you must specify the -u  option
       to quote Unicode using standard U+XXXX notation.

       Currently  the  file  name of a file containing known hashes may not be
       specified as a unicode filename, but you can specify the name using tab
       completition  or an asterisk (e.g. md5deep -m *.txt where there is only
       one file with a .txt extension).


RETURN VALUE
       Returns a bit-wise value based on the success of the operation and  the
       status of any matching operations.

       0      Success.  Note that the program considers itself successful even
              when it encounters read errors,  permission  denied  errors,  or
              finds directories when not in recursive mode.

       1      Unused  hashes.  Under  any  of the matching modes, returns this
              value if one or more of the known hashes was not matched by  any
              of the input files.

       2      Unmatched  inputs. Under any of the matching modes, returns this
              value if one or more of the input values did not  match  any  of
              the known hashes.

       64     User  error,  such  as  trying  to do both positive and negative
              matching at the same time.

       128    Internal error, such as memory  corruption  or  uncaught  cycle.
              All internal errors should be reported to the developer! See the
              section "Reporting Bugs" below.



AUTHOR
       md5deep was written by Jesse Kornblum,  research@jessekornblum.com  and
       Simson Garfinkel.


KNOWN ISSUES
       Using  the -r flag cannot be used to recursively process all files of a
       given extension in a directory. This is a feature, not a bug.   If  you
       need to do this, use the find(1) command.


REPORTING BUGS
       We  take  all  bug reports very seriously. Any bug that jeopardizes the
       forensic integrity of this program could have serious  consequences  on
       people's lives. When submitting a bug report, please include a descrip‐
       tion of the problem, how you found it, and your contact information.

       Send bug reports to the author at the address above.


COPYRIGHT
       This program is a work of the US Government. In accordance with 17  USC
       105,  copyright protection is not available for any work of the US Gov‐
       ernment.  This program is PUBLIC DOMAIN. Portions of this program  con‐
       tain  code  that  is  licensed  under  the  terms of the General Public
       License (GPL).  Those portions  retain  their  original  copyright  and
       license. See the file COPYING for more details.

       There  is NO warranty for this program; not even for MERCHANTABILITY or
       FITNESS FOR A PARTICULAR PURPOSE.


SEE ALSO
       More information and installation instructions  can  be  found  in  the
       README  file.  Current  versions  of both documents can be found on the
       project homepage: http://md5deep.sourceforge.net/

       The MD5 specification, RFC 1321, is available at
       http://www.ietf.org/rfc/rfc1321.txt

       The SHA-1 specification, RFC 3174, is available at
       http://www.faqs.org/rfcs/rfc3174.html

       The SHA-256 specification, FIPS 180-2, is available at
       http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf

       The SHA-3-256 specification is available at
       http://keccak.noekeon.org/

       The Tiger specification is available at
       http://www.cs.technion.ac.il/~biham/Reports/Tiger/

       The Whirlpool specification is available at
       http://planeta.terra.com.br/informatica/paulobarreto/WhirlpoolPage.html



AFOSI                         v4.4 - 29 Jan 2014                    MD5DEEP(1)
