dd (Unix)

From Wikipedia, the free encyclopedia

dd is a common UNIX program whose primary purpose is the low-level copying and conversion of raw data. dd is an abbreviation for "data definition" in IBM JCL, and the command's syntax is meant to be reminiscent of this.^[1]

dd is used to copy a specified number of bytes or blocks, performing on-the-fly byte order conversions, as well as more esoteric EBCDIC to ASCII conversions.^[2] dd can also be used to copy regions of raw device files, e.g. backing up the boot sector of a hard disk, or to read fixed amounts of data from special files like /dev/zero or /dev/random.^[3]

It can also be used in computer forensics when the magnetic pattern of an entire disk needs to be preserved as a byte-exact copy. Using cp would not be possible because data from deleted files still physically present on a disk are not visible through the file system interface.

It is jokingly said to stand for "data destroyer" or "delete data", since, being used for low-level operations on hard disks, a small mistake, such as reversing the if and of parameters, may accidentally render the entire disk unusable.^[2]

[edit] Usage

The command line syntax of dd is significantly different from most other UNIX programs, and because of its ubiquity it is resistant to recent attempts to enforce a common syntax for all command line tools. Generally, dd uses an option=value format, whereas most Unix programs use a -option value format. Also, dd's input is specified using the "if" (input file) option, while most programs simply take the name by itself. It is rumored to have been based on IBM's JCL, and though the syntax may have been a joke^[1], there seems never to have been any effort to write a more Unix-like replacement.

Example use of dd command to create an ISO disk image from a CD-ROM:

dd if=/dev/cdrom of=/home/sam/myCD.iso bs=2048 conv=sync,notrunc

Note that an attempt to copy the entire disk image using cp may omit the final block if it is an unexpected length; dd will always complete the copy if possible.

Using dd to wipe an entire disk with random data (though there are better tools for this, like Darik's Boot and Nuke):

dd if=/dev/urandom of=/dev/hda

alternative:

for n in `seq 7`; do dd if=/dev/urandom of=/dev/sda bs=8b conv=notrunc; done

Using dd to duplicate one hard disk partition to another hard disk:

dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=notrunc,noerror

Note that notrunc means do not truncate the output file.^{[clarification needed]} Noerror means to keep going if there is an error (though a better tool for this would be ddrescue).

To duplicate a disk partition as a disk image file on a different partition

 dd if=/dev/sdb2 of=/home/sam/partition.image bs=4096 conv=notrunc,noerror

Create a 1GB file containing only zeros (bs=blocksize, count=number of blocks):

 dd if=/dev/zero of=mytestfile.out bs=1000 count=1000000

To make sure that my drive is really zeroed out!!

 dd if=/dev/sda | hexdump -C | grep [^00]

To duplicate the first 2 sectors of the floppy.

 dd if=/dev/fd0 of=/home/sam/MBRboot.image bs=512 count=2

To duplicate master boot record only

 dd if=/dev/sda of=/home/sam/MBR.image bs=446 count=1

To making drive benchmark test and analyzing read and write performance

 dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file
 dd if=/home/sam/1Gb.file bs=64k | dd of=/dev/null

To make a file of 100 random bytes:

 dd if=/dev/urandom of=/home/sam/myrandom bs=100 count=1

To convert a file to uppercase:

 dd if=filename of=filename conv=ucase

To search the system memory:

 dd if=/dev/mem | hexdump -C | grep 'some-string-of-words-in-the-file-you-forgot-to-save-before-the-power-failed'

Image a partition to another machine:

 On source machine:
   dd if=/dev/hda bs=16065b | netcat < targethost-IP > 1234
 On target machine:
   netcat -l -p 1234 | dd of=/dev/hdc bs=16065b

Everybody has mentioned the first obvious fix: raise your blocksize from the default 512 bytes. The second fix addresses the problem that with a single dd, you are either reading or writing. If you pipe the first dd into a second one, it'll let you run at the max speed of the slowest device.

 dd if=/dev/ad2 conv=noerror,sync bs=64k | dd of=/dev/ad3 bs=64k

Sending a USR1 signal to a running `dd' process makes it print I/O statistics to standard error and then resume copying.

 $ dd if=/dev/zero of=/dev/null& pid=$!
 $ kill -USR1 $pid 
 18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB) copied, 
 34.6279 seconds, 271 MB/s 

Create a 1GB sparse file or resize an existing file to 1GB without overwriting:

 dd if=/dev/zero of=mytestfile.out bs=1 count=0 seek=1G

Some implementations understand x as a multiplication operator in the block size and count parameters:

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image

where the "b" suffix indicates that the units are 512-byte blocks. Unix block devices use this as their allocation unit by default.

For the value of bs field, following decimal number can be suffixed:

 w means 2
 b means 512
 k means 1024

Hence bs=2*80*18b means, 2*80*18*512=1474560 which is the exact size of 1440 KiB floppy disk

To mount that image mount -o loop floppy.image /mntpoint

[edit] Output messages

The GNU variant of dd as supplied with Linux does not describe the format of the messages displayed on stdout on completion, however these are described by other implementations e.g. that with BSD.

Each of the "Records in" and "Records out" lines shows the number of complete blocks transferred + the number of partial blocks, e.g. because the physical medium ended before a complete block was read.

[edit] Recovery-oriented variants of dd

Open Source unix-based programs for rescue include dd_rescue and dd_rhelp, which work together, or GNU ddrescue.

Antonio Diaz Diaz (the developer of GNU ddrescue) compares^[4] the variants of dd for the task of rescuing:

The standard utility dd does a linear read of the drive, so it can take a long time or even fry the drive without rescueing anything if the errors are at the beginning of the drive. Kurt Garloff's dd_rescue does basically the same thing as dd, only more efficiently. LAB Valentin's dd_rhelp is a complex shell script that runs Garloff's dd_rescue many times, trying to be strategic about copying the drive, but it is very inefficient.

• dd_rhelp first extracts all the readable data, and saves it to a file, inserting zeros where bytes cannot be read. Then it tries to re-read the invalid data and update this file.
• GNU ddrescue can be used to copy data directly to a new disk if needed, just like Linux dd.

dd_rhelp or GNU ddrescue will yield a complete disk image, faster but possibly with some errors. GNU ddrescue is generally much faster, as it is written entirely in C++, whereas dd_rhelp is a shell script wrapper around dd_rescue. Both dd_rhelp and GNU ddrescue aim to copy data fast where there are no errors, then copy in smaller blocks and with retries where there are errors. GNU ddrescue is easy to use with default options, and can easily be downloaded and compiled on Linux-based Live CDs such as Knoppix, and can be used with SystemRescueCD.

GNU ddrescue example ^[5]

# first, grab most of the error-free areas in a hurry:
ddrescue -n /dev/old_disk /dev/new_disk rescued.log
# then try to recover as much of the dicey areas as possible:
ddrescue -r 1 /dev/old_disk /dev/new_disk rescued.log

[edit] See also

• List of Unix programs
• Backup
• Disk cloning
• Disk image
• Loopback
• SpinRite (article includes coverage of recovery-oriented variants of dd)

[edit] References

[edit] External links

• dd: manual page from GNU coreutils.
• dd for Windows.
• The Open Group Base Specifications Issue 6 of the dd utility
• GNU ddrescue.
• Manual for GNU ddrescue.
• Softpanorama dd page.
• Learn The DD Command.