tar (file format)

From Wikipedia, the free encyclopedia

Jump to: navigation, search
tar
Image:Torchlight tar.png

GNU tar 1.16 showing three common types of Tarballs (shown in red).
Filename extension .tar
Internet media type application/x-tar
Uniform Type Identifier public.tar-archive
Magic number ustar at byte 257
Type of format file archiver
Contained by gzip, bzip2, lzma, lzip, compress

In computing, tar (derived from tape archive) is both a file format (in the form of a type of archive bitstream) and the name of the program used to handle such files. The format was created in the early days of Unix and standardized by POSIX.1-1988 and later POSIX.1-2001. Initially developed to be written directly to sequential I/O devices for tape backup purposes, it is now commonly used to collect many files into one larger file, for distribution or archiving, while preserving file system information such as user and group permissions, dates, and directory structures.

Contents

[edit] Compression and naming

Conventionally, uncompressed tar archive files have names ending in ".tar". Unlike more recent archive formats, tar files (somefile.tar) are commonly compressed as a whole rather than piecemeal. Applying a compression utility such as gzip, bzip2, lzma or compress to a tar file produces a compressed tar file, typically named with an extension indicating the type of compression (e.g.: somefile.tar.gz).

Popular tar programs like the BSD and GNU versions of tar support the command line options -z (gzip), and -j (bzip2) to automatically compress or decompress the archive file it is currently working with. GNU tar from version 1.20 onwards also supports --lzma (LZMA). Both will automatically extract compressed gzip and bzip2 archives with or without these options.

MS-DOS's 8.3 filename limitations, and a desire for brevity, resulted in additional popular conventions for naming compressed tar archives:

  • ".tgz" is equivalent to ".tar.gz".
  • ".tbz" (or ".tbz2" or ".tb2") is equivalent to ".tar.bz2".
  • ".taz" is equivalent to ".tar.Z",

A tar file or compressed tar file is commonly referred to as a tarball.

[edit] Format details

A tar file is the concatenation of one or more files. Each file is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes and the extra space is zero filled. The end of an archive is marked by at least two consecutive zero-filled records. (The origin of tar's record size appears to be the 512-byte disk sectors used in the Version 7 Unix file system.)

Many historic tape drives read and write variable-length data blocks, leaving significant wasted space on the tape between blocks (for the tape to physically start and stop moving). Some tape drives (and raw disks) only support fixed-length data blocks. Also, when writing to any medium such as a filesystem or network, it takes less time to write one large block than many small blocks. Therefore, the tar command writes data in blocks of many 512 byte records. The user can specify a blocking factor, which is the number of records per block; the default is 20, producing 10 kilobyte blocks (which was large when UNIX was invented, but now seems rather small). The final block of an archive is padded out to full length with zero bytes.

[edit] File header

The file header block contains metadata about a file. To ensure portability across different architectures with different byte orderings, the information in the header block is encoded in ASCII. Thus if all the files in an archive are text files, and have ASCII names, then the archive is essentially an ASCII text file (containing many NUL or all-zero characters). Subsequent extensions have diluted this original attribute (which was never particularly valuable, due to the presence of NULs, which are often poorly handled by text-manipulation programs).

The fields defined by the original Unix tar format are listed in the table below. The link indicator/file type table includes some modern extensions. When a field is unused it is filled with NUL bytes. The header is padded with NUL bytes to make it fill a 512 byte block.

Field Offset Field Size Field
0 100 File name
100 8 File mode
108 8 Owner's numeric user ID
116 8 Group's numeric user ID
124 12 File size in bytes
136 12 Last modification time in numeric Unix time format
148 8 Checksum for header block
156 1 Link indicator (file type)
157 100 Name of linked file
Link indicator field
Value Meaning
'0' Normal file
(ASCII NUL) Normal file (now obsolete)
'1' Hard link
'2' Symbolic link
'3' Character special
'4' Block special
'5' Directory
'6' FIFO

A directory is also indicated by having a trailing slash (/) in the name.

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final null or space character should be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, some versions of tar, including the GNU implementation, support an extension in which the high-order bit of the leftmost byte of a numeric field indicates that the number is encoded in big-endian binary. Additionally, versions of tar from 1999 and before pad the values with spaces instead of zeroes.

The checksum is calculated by taking the sum of the unsigned byte values of the header block with the eight checksum bytes taken to be ascii spaces (decimal value 32). It is stored as a six digit octal number with leading zeroes followed by a null and then a space. Various implementations do not adhere to this, so relying on the first white space trimmed six digits for checksum yields better compatibility. In addition, some historic tar implementations treated bytes as signed. Readers must calculate the checksum both ways, and treat it as good if either the signed or unsigned sum matches the included checksum.

Unix filesystems support multiple links (names) for the same file. If several such files appear in a tar archive, only the first one is archived as a normal file; the rest are archived as hard links, with the "name of linked file" field set to the first one's name. On extraction, such hard links should be recreated in the file system.

[edit] UStar format

Most modern tar programs read and write archives in the new UStar (Uniform Standard Tape Archive) format, which has an extended header definition as defined by the POSIX (IEEE P1003.1) standards group. Older tar programs will ignore the extra information, while newer programs will test for the presence of the "ustar" string to determine if the new format is in use. The UStar format allows for longer file names and stores extra information about each file.

Field Offset Field Size Field
0 156 (as in old format)
156 1 Type flag
157 100 (as in old format)
257 6 UStar indicator "ustar"
263 2 UStar version "00"
265 32 Owner user name
297 32 Owner group name
329 8 Device major number
337 8 Device minor number
345 155 Filename prefix

[edit] Example

The example below shows the ASCII dump of a header block from a tar file created using the GNU tar program. It was dumped with the od program. The "ustar" magic string followed by two spaces can be seen, meaning that the tar file is in GNU format, partially incompatible with the true UStar standard (in POSIX.1-1988), which has the signature "ustar" followed by a NUL character.[1]

0000000   e   t   c   /   p   a   s   s   w   d nul nul nul nul nul nul
0000020 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0000140 nul nul nul nul   0   1   0   0   6   4   4 nul   0   0   0   0
0000160   0   0   0 nul   0   0   0   0   0   0   0 nul   0   0   0   0
0000200   0   0   4   1   3   5   5 nul   1   0   1   5   5   0   6   1
0000220   1   0   5 nul   0   1   1   5   5   6 nul  sp   0 nul nul nul
0000240 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0000400 nul   u   s   t   a   r  sp  sp nul   r   o   o   t nul nul nul
0000420 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000440 nul nul nul nul nul nul nul nul nul   r   o   o   t nul nul nul
0000460 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0001000

[edit] Tarbombs

A tarbomb is derogatory hacker slang used to refer to a tarball that does not follow the usual conventions, i.e. it contains many files that extract into the working directory. Such a tarball can create problems by overwriting files of the same name in the working directory, or mixing one project's files into another. It is almost always an inconvenience to the user, who is obliged to identify and delete a number of files scattered throughout the directory's contents. Such behavior is considered bad etiquette on the part of the archive's creator.

A related problem is the use of absolute paths or parent directory references when creating tarballs. Files extracted from such tarballs will often be created in unusual locations outside the working directory, and like a tarbomb have the potential to overwrite existing files. GNU tar by default refuses to create or extract absolute paths, but is still vulnerable to parent-directory references.

A user can avoid both of these problems by first examining a tar file with a command like tar -tf archive.tar. This does not extract any files, but displays the names of all files in the archive. If any are problematic, the user can create a new empty directory and extract the tarball into it -- or avoid the tarball entirely.

[edit] Tarpipe

A tarpipe is the process of creating a tar archive in stdout and then, in another directory, extracting the tar file from the piped stdin. This is a useful way to copy directories and subdirectories, especially if the directories contain special files, such as symlinks, and character or block devices.

 tar -cf - "${srcdir}" | ( cd "${destdir}" && tar -xvf - )

[edit] Use in software distribution

The most common publication format for the released source code of free software is a tarball. For example, the GNU Project published a version of Emacs as "emacs-22.3.tar.gz" in September 2008. This use of the common tar format (and publication of the tarball on the Internet via http or ftp) masks the varying details of each project's source code revision control system.

[edit] See also

[edit] Notes

  1. ^ The OpenBSD 3.7 tar does not have the 2 space characters after ustar. They are nul characters. GNU tar by default creates incompatible archives in case that the archives contain path names that are longer than 100 characters and GNU tar writes an incorrect size field in case a sparse file has more than 4 holes.

[edit] External links

Personal tools