# MAR Files

MAR files, short for Mozilla ARchive files, are integral to Firefox Application
Update, so this page will document what they are and go into their structure in
some detail.

## Overview

A MAR file consists of headers, signatures, multiple files including at least
one manifest, and an index. Only one manifest, of the newest supported version,
will be used; others will be ignored.

The manifest must be a complete file, but others may be patch files. Patch files
are used to transform the old version of the file to the new version. Currently
supported patch algorithms are bspatch and Zucchini.

Files are optionally compressed. Currently supported compression algorithms are
bzip2 and xz.

## Types of MARs

There are two types of MAR files: partial MARs and complete MARs.

A complete MAR contains an entire installation. It does not require any
particular files to be on disk in order to update. Theoretically, it can be used
to update from any version, but
[Watershed Updates](../../../../update-infrastructure/index.md#watershed-updates)
complicate this somewhat.

A partial MAR contains only the changes between one version and another. While
they may contain some files in their entirety, they mostly have binary patches
that can be used to update an existing file to the new version. The advantage of
this is that partial MARs can be much smaller than complete MARs. But the
disadvantage is that a partial MAR must target a specific version of Firefox so
that we can provide the correct patches to update that version of the files. And
we just cannot provide a partial targeting every version of Firefox. We
currently generate partials for the previous four versions released (including
point versions). When Firefox is more out-of-date than that, a complete MAR must
be used. Another limitation is that the files on the disk must not have been
changed. We verify this by including headers in the patch file with the expected
size and CRC of the existing file. If these do not match the file on disk, the
update fails and we fall back to using a complete MAR.

## Manifest

The manifest is a special file included in the MAR that describes the contents
of the MAR and how to install it. It is identified by its filename:
`updatev3.manifest`.

Empty lines and `#`-prefixed comment lines are allowed. Otherwise, each line of
the manifest will consist of a method name followed by variable number of quoted
parameters. These are separated by an arbitrary amount of space (`0x20`) and/or
tab (`0x09`) characters. The lines must end with a Windows/CRLF newline
(`0x0D0A`).

Here is an example manifest containing each of the possible method types:

```
type "partial"
add "2/20/20text0"
add-if "distribution/extensions/extensions0" "distribution/extensions/extensions0/extensions0text0"
add-if-not "defaults/pref/channel-prefs.js" "defaults/pref/channel-prefs.js"
patch "0/0exe0.exe.patch" "0/0exe0.exe"
patch-if "distribution/extensions/extensions1" "distribution/extensions/extensions1/extensions1png1.png.patch" "distribution/extensions/extensions1/extensions1png1.png"
remove "1/10/10text0"
rmdir "9/99/"
rmrfdir "9/98/"
```

`type` is special method that should be `partial` or `complete` depending on
[what kind of MAR this is](#types-of-mars). If included, it must be the first
method specified. The rest are implemented by classes deriving from `Action` in
`/toolkit/mozapps/update/updater/updater.cpp`.

## Manipulation

The recommended way to manually manipulate (create, extract, sign, etc) MAR
files is with the tool located
[here](https://github.com/mozilla-releng/build-mar).

## MAR Structure

This section was almost directly copied from
[the wiki](https://wiki.mozilla.org/Software_Update:MAR) in order to consolidate
our documentation into a place where it is more likely to be updated. This
document should be considered more up-to-date than that one.

### Why the bespoke structure?

This question was given a fair amount of consideration. Ultimately, we decided
to go with a custom file format because using libjar would have required a fair
bit of hacking. Writing custom code was a simpler option, and it resulted in
less code (mar_read.c is less than 300 lines of code). Moreover, the update
system does not need a standard file format. The elements stored in the archive
are bzip2 compressed binary diffs, generated using a variation of bsdiff. So,
being able to unpack the archive file using standard tools wouldn't be very
useful in and of itself.

### Byte Ordering

All fields are in big-endian format. The signatures are in
NSS / OpenSSL / big-endian order and not CryptoAPI order. If CryptoAPI is used
to check a signature, the bytes of the signature must be reversed before
verifying the signature using CryptVerifySignature.

### Constraints

To protect against invalid inputs the following constraints are in place:

- There are at most 8 signatures.
- The file size of the MAR file is at most 500MB.
- No signature is more than 2048 bytes long.

### File Layout

The following sections, in the given order, make up the structure of a MAR
file. Some of these sections have been added over time and thus may not be
present in sufficiently old MAR files. Old parsers will completely ignore these
sections as they will simply appear to be unaddressed space in the "Files"
section. Modern updater binaries will, however, reject MAR files without
signatures.

#### Header

1. (4 bytes) MAR identifier which must be `MAR1`.
2. (4 bytes) Byte offset of the file index header relative to the start of the
   file.
3. (8 bytes) Size in bytes of the entire MAR file.
4. (4 bytes) Number of signatures (minimum of 0, maximum of 8).

#### Signatures

This section will be repeated as many times as indicated by the relevant header.
Note that these signatures must sign all bytes of the MAR file excluding the
signatures themselves.

1. (4 bytes) ID representing the type of signature algorithm.
2. (4 bytes) Size in bytes of the signature that follows.
3. The signature itself.

#### Additional Sections Header

This consists of 4 bytes representing the number of additional sections.

#### Additional Sections

There will be as many of these sections as indicated in the Additional Sections
Header, immediately prior.

1. (4 bytes) Section size, in bytes. Note that, unlike the signature size, this
   is the size of the section and its metadata (the size and identifier).
2. (4 bytes) Section identifier.
3. The rest of the section. Consists of 8 bytes fewer than the section size
   given above.

#### Files

All file data. Must be addressed by the index, below. The length will be the sum
of content sizes of all files in the index.

#### Index header

This consists of 4 bytes representing the size of the index in bytes.

#### Index entry

This section will be repeated to fill the size indicated by the index header,
describing all files contained in the MAR.

1. (4 bytes) Byte offset of the file, relative to the start of the MAR.
2. (4 bytes) The content size in bytes.
3. (4 bytes) File permission bits, in standard unix-style format.
4. Variable length filename
5. (1 byte) Null terminator

### Additional Sections

These are meant to be flexible, so more could potentially be added later.

#### Product Information

The product information section identifies the product and channel this MAR file
applies to. It also includes the new version number to avoid downgrades.

This section has this structure:

1. (4 bytes) The size of the product information block.
2. (4 bytes) The section ID, `1` in this case.
3. (<64 bytes) The product and channel (such as from MAR_CHANNEL_ID).
   Example: mozilla-central
4. (1 byte) Null terminator.
5. (<32 bytes) The product version string (such as from MOZ_APP_VERSION)
   Examples: 12.0.1.5371, 12.0a1
6. (1 byte) Null terminator.

## Modifying Test MARs

Occasionally we need to modify a MAR file that we use in testing. These are some
modifications that have been needed before and the steps to accomplish them.

### Rename Files

In this example, we will rename some of the files in `partial_mac.mar`.

Regarding signing, note the following useful things: (a) The test MAR signatures
successfully verify against
`toolkit/mozapps/update/updater/xpcshellCertificate.der` and (b) according to
[this comment](https://searchfox.org/mozilla-central/rev/fc00627e34639ef1014e87d9fa24091905e9dc5d/toolkit/mozapps/update/updater/moz.build#41-43),
that certificate was generated from `mycert` in
`modules/libmar/tests/unit/data`.

1.  `./mach build` so that `<obj>/dist/bin/signmar` is available.
2.  Install the MAR manipulation utility, if necessary: `pip install mar`.
3.  Make a temporary working directory:
    `cd /path/to/mozilla/repo && mkdir temp && cd temp`
4.  Extracted the MAR to be changed:
    `mar -J -x ../toolkit/mozapps/update/tests/data/partial_mac.mar`.
    The `-J` specifies a compression type of `xz`. You can instead specify
    `--auto` to automatically detect the compression type (though you may want
    to know the original compression later for recompression). You can check
    the compression type by running
    `mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar` and looking for
    the `Compression type:` line.
5.  Rename extracted files, as necessary.
6.  Edit `updatev2.manifest` and `updatev3.manifest` to update the changed paths.
7.  Run `mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar` to get a
    complete list of the files originally in that MAR as well as the
    product/version and channel strings (in this case `xpcshell-test` and `*`
    respectively).
8.  Create the new MAR:
    `mar -J -c partial_mac_unsigned.mar -V '*' -H xpcshell-test <file1> <file2> ...`,
    individually specifying each file path listed in by `mar -T`, substituting
    with renamed paths as necessary.
9.  Sign the MAR:
    `../<obj>/dist/bin/signmar -d ../modules/libmar/tests/unit/data -n mycert -s partial_mac_unsigned.mar partial_mac.mar`.
10. To verify signing:
    `../<obj>/dist/bin/signmar -D ../toolkit/mozapps/update/updater/xpcshellCertificate.der -v partial_mac.mar`.
    This appears to output nothing on success, but it's probably good to check
    to make sure `echo $?` displays `0`. I also compared the output of
    `mar -T partial_mac.mar` to that of the original.
11. To verify files: Extract the new MAR with
    `mkdir cmp && cd cmp && mar -J -x ../partial_mac.mar && cd ..` and verify
    the files match the originals.
12. Overwrite the original MAR with the new one and remove the `temp`
    directory:
    `cd .. && mv -f temp/partial_mac.mar toolkit/mozapps/update/tests/data/partial_mac.mar && rm -rf temp`

### Generate Zucchini Partials

In this example, we will generate partial MARs using Zucchini rather than
bspatch.

First, note that in `partial.mar`, all exe patches are the same patch (turning
`complete.exe` into `partial.exe`), and all png patches are the same patch
(turning `complete.png` into `partial.png`):

```
$ mar -J -x ../partial.mar

$ grep -r MBDIFF10 .
Binary file ./0/00/00png0.png.patch matches
Binary file ./0/0exe0.exe.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
Binary file ./exe0.exe.patch matches
Binary file ./searchplugins/searchpluginspng0.png.patch matches
Binary file ./searchplugins/searchpluginspng1.png.patch matches

$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./0/00/00png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng1.png.patch

$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
bd6e413d3248cfbeff65e104b6c4cd39 *./0/0exe0.exe.patch
bd6e413d3248cfbeff65e104b6c4cd39 *./exe0.exe.patch
```

Generate the equivalent zucchini patches:

```
$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.exe toolkit/mozapps/update/tests/data/partial.exe exe0.exe.patch -keep
$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.png toolkit/mozapps/update/tests/data/partial.png 00png0.png.patch -keep
```

Replace all patch files on disk by their Zucchini equivalent, then double check
by rerunning the `md5sum` commands:

```
$ grep -r Zucc .
Binary file ./0/00/00png0.png.patch matches
Binary file ./0/0exe0.exe.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
Binary file ./exe0.exe.patch matches
Binary file ./searchplugins/searchpluginspng0.png.patch matches
Binary file ./searchplugins/searchpluginspng1.png.patch matches

$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
958bae1b40904145959ba45f988a7156 *./0/00/00png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png1.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png1.png.patch
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng0.png.patch
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng1.png.patch

$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
7750e88e0c1d006710abbd625710748b *./0/0exe0.exe.patch
7750e88e0c1d006710abbd625710748b *./exe0.exe.patch
```

Generate an unsigned mar file. Unfortunately on Windows this doesn't preserve
the permissions from the original mar. The simplest solution is to do this part
on a UNIX (file)system.

```
$ mar -J -c ../partial_zucchini_unsigned.mar -H xpcshell-test -V '*' 2/20/20png0.png 0/0exe0.exe.patch 0/00/00text2 distribution/extensions/extensions0/extensions0text0 distribution/extensions/extensions0/extensions0png1.png.patch 0/00/00text0 distribution/extensions/extensions0/extensions0png0.png.patch searchplugins/searchpluginspng1.png.patch distribution/extensions/extensions1/extensions1png0.png.patch distribution/extensions/extensions1/extensions1text0 searchplugins/searchpluginstext0 distribution/extensions/extensions1/extensions1png1.png.patch defaults/pref/channel-prefs.js update-settings.ini updatev2.manifest searchplugins/searchpluginspng0.png.patch precomplete 0/00/00png0.png.patch updatev3.manifest exe0.exe.patch 2/20/20text0
```

Generate a signed mar from the unsigned mar:

```
$ /d/mozilla-source/firefox/obj-x86_64-pc-windows-msvc/dist/bin/signmar.exe -d /d/mozilla-source/firefox/modules/libmar/tests/unit/data/ -n mycert -s ../partial_zucchini_unsigned.mar ../partial_zucchini.mar
```