Vendoring Third Party Components

The firefox source tree vendors many third party dependencies. The build system provides a normalized way to keep track of:

1. The upstream source license, location and revision

2. (Optionally) The upstream source modification, including

   1. Mozilla-specific patches

   2. Custom update actions, such as excluding some files, moving files around etc.

This is done through a descriptive moz.yaml file added to the third party sources, and the use of:

./mach vendor [options] ./path/to/moz.yaml

to interact with it.

Template moz.yaml file

---
# Third-Party Library Template
# All fields are mandatory unless otherwise noted

# Version of this schema
schema: 1

bugzilla:
  # Bugzilla product and component for this directory and subdirectories
  product: product name
  component: component name

# Document the source of externally hosted code
origin:

  # Short name of the package/library
  name: name of the package

  description: short (one line) description

  # Full URL for the package's homepage/etc
  # Usually different from repository url
  url: package's homepage url

  # Human-readable identifier for this version/release
  # Generally "version NNN", "tag SSS", "bookmark SSS"
  release: identifier

  # Revision to pull in
  # Must be a long or short commit SHA (long preferred)
  revision: sha

  # The package's license, where possible using the mnemonic from
  # https://spdx.org/licenses/
  # Multiple licenses can be specified (as a YAML list)
  # A "LICENSE" file must exist containing the full license text
  license: MPL-2.0

  # If the package's license is specified in a particular file,
  # this is the name of the file.
  # optional
  license-file: COPYING

  # If there are any mozilla-specific notes you want to put
  # about a library, they can be put here.
  notes: Notes about the library

# Configuration for the automated vendoring system.
# optional
vendoring:

  # Repository URL to vendor from
  # eg. https://github.com/kinetiknz/nestegg
  # Any repository host can be specified here, however initially we'll only
  # support automated vendoring from selected sources.
  url: source url (generally repository clone url)

  # Type of hosting for the upstream repository
  # Valid values are 'gitlab', 'github', googlesource
  source-hosting: gitlab

  # Type of Vendoring
  # This is either 'regular', 'individual-files', or 'rust'
  # If omitted, will default to 'regular'
  flavor: rust

  # Type of git reference (commit, tag) to track updates from.
  # You cannot use tag tracking with the individual-files flavor
  # If omitted, will default to tracking commits.
  tracking: commit

  # When using tag tracking (only on Github currently) use a release artifact
  # for the source code instead of the automatically built git-archive exports.
  # The source repository must build these artifacts with consistent filenames
  # for every tag. This is useful when the Github repository uses submodules
  # since they are not included in the git-archives.
  # Substitution is performed on the filename, {tag} is replaced with the tag name.
  # optional
  release-artifact: "rnp-{tag}.tar.gz"

  # Base directory of the location where the source files will live in-tree.
  # If omitted, will default to the location the moz.yaml file is in.
  vendor-directory: third_party/directory

  # Allows skipping certain steps of the vendoring process.
  # Most useful if e.g. vendoring upstream is complicated and should be done by a script
  # The valid steps that can be skipped are listed below
  skip-vendoring-steps:
    - fetch
    - keep
    - include
    - exclude
    - move-contents
    - hg-add
    - spurious-check
    - update-moz-yaml
    - update-moz-build

  # List of patch files to apply after vendoring. Applied in the order
  # specified, and alphabetically if globbing is used. Patches must apply
  # cleanly before changes are pushed.
  # Patch files should be relative to the vendor-directory rather than the gecko
  # root directory.
  # All patch files are implicitly added to the keep file list.
  # optional
  patches:
    - file
    - path/to/file
    - path/*.patch
    - path/**  # Captures all files and subdirectories below path
    - path/*   # Captures all files but _not_ subdirectories below path. Equivalent to `path/`

  # List of files that are not removed from the destination directory while vendoring
  # in a new version of the library. Intended for mozilla files not present in upstream.
  # Implicitly contains "moz.yaml", "moz.build", and any files referenced in
  # "patches"
  # optional
  keep:
    - file
    - path/to/file
    - another/path
    - *.mozilla

  # Files/paths that will not be vendored from the upstream repository
  # Implicitly contains ".git", and ".gitignore"
  # optional
  exclude:
    - file
    - path/to/file
    - another/path
    - docs
    - src/*.test

  # Files/paths that will always be vendored from source repository, even if
  # they would otherwise be excluded by "exclude".
  # optional
  include:
    - file
    - path/to/file
    - another/path
    - docs/LICENSE.*

  # Files that are modified as part of the update process.
  # To avoid creating updates that don't update anything, ./mach vendor will detect
  # if any in-tree files have changed. If there are files that are always changed
  # during an update process (e.g. version numbers or source revisions), list them
  # here to avoid having them counted as substative changes.
  # This field does NOT support directories or globbing
  # optional
  generated:
    - '{yaml_dir}/vcs_version.h'

  # If neither "exclude" or "include" are set, all files will be vendored
  # Files/paths in "include" will always be vendored, even if excluded
  # eg. excluding "docs/" then including "docs/LICENSE" will vendor just the
  #     LICENSE file from the docs directory

  # All three file/path parameters ("keep", "exclude", and "include") support
  # filenames, directory names, and globs/wildcards.

  # Actions to take after updating but before applying patches. Applied in order.
  # The action subfield is required. It must be one of:
  #   - copy-file
  #   - move-file
  #   - move-dir
  #   - replace-in-file
  #   - replace-in-file-regex
  #   - delete-path
  #   - run-script
  #   - vcs-add-remove-files
  # Unless otherwise noted, all subfields of action are required.
  #
  # If the action is copy-file, move-file, or move-dir:
  #   from is the source file
  #   to is the destination
  #
  # If the action is replace-in-file or replace-in-file-regex:
  #   pattern is what in the file to search for. It is an exact strng match.
  #   with is the string to replace it with. Accepts the special keyword
  #     '{revision}' for the commit we are updating to.
  #   File is the file to replace it in.
  #
  # If the action is vcs-add-remove-files
  #   path is the directory to add all new files to version control, and
  #     delete all removed files.
  #
  # If the action is delete-path
  #   path is the file or directory to recursively delete
  #
  # If the action is run-script:
  #   script is the script to run
  #   cwd is the directory the script should run with as its cwd
  #   args is a list of arguments to pass to the script
  #   Note: 'script', 'cwd', and entries in 'args' support path placeholders
  #     '{vendor_dir}', '{yaml_dir}', '{tmpextractdir}', '{topsrcdir}'.
  #     For 'run-script'/'run-command', '{cwd}' is also supported but must
  #     appear only at the beginning of a path. In 'args', the literal
  #     '{revision}' will be replaced with the target revision string.
  #
  # If the action is run-command:
  #   command is the command to run
  #      Unlike run-script, `command` is _not_ processed to be relative
  #      to the vendor directory, and is passed directly to python's
  #      execution code without any path substitution or manipulation
  #   cwd is the directory the command should run with as its cwd
  #   args is a list of arguments to pass to the command
  #
  # Unless specified otherwise, all files/directories are relative to the
  #     vendor-directory. The following placeholders are supported in most
  #     path fields: '{vendor_dir}', '{yaml_dir}', '{tmpextractdir}', '{topsrcdir}'.
  #     If the vendor-directory differs from the yaml directory, '{yaml_dir}'
  #     makes the path relative to the yaml directory.
  #     For 'run-script'/'run-command', '{cwd}' is additionally supported and
  #     must only be used at the beginning of a path.
  #
  # optional
  update-actions:
    - action: copy-file
      from: include/vcs_version.h.in
      to: '{yaml_dir}/vcs_version.h'

    - action: replace-in-file
      pattern: '@VCS_TAG@'
      with: '{revision}'
      file: '{yaml_dir}/vcs_version.h'

    - action: delete-path
      path: '{yaml_dir}/config'

    - action: run-script
      script: '{cwd}/generate_sources.sh'
      cwd: '{yaml_dir}'

  # Actions to take after patches have been applied. Applied in order.
  # Uses the same action types as update-actions.
  # optional
  post-patch-actions:
    - action: run-script
      script: '{yaml_dir}/post_patch_script.py'
      cwd: '{yaml_dir}'
      args: ['{revision}']


# Configuration for automatic updating system.
# optional
updatebot:

  # TODO: allow multiple users to be specified
  # Phabricator username for a maintainer of the library, used for assigning
  # reviewers. For a review group, preface with #, such as "#build""
  maintainer-phab: tjr

  # Bugzilla email address for a maintainer of the library, used for needinfos
  maintainer-bz: tom@mozilla.com

  # Optional: A preset for ./mach try to use. If present, fuzzy-query and fuzzy-paths will
  # be ignored. If it, fuzzy-query, and fuzzy-path are omitted, ./mach try auto will be used
  try-preset: media

  # Optional: A query string for ./mach try fuzzy. If try-preset, it and fuzzy-paths are omitted
  # then ./mach try auto will be used
  fuzzy-query: media

  # Optional: An array of test paths for ./mach try fuzzy. If try-preset, it and fuzzy-query are
  # omitted then ./mach try auto will be used
  fuzzy-paths: ['media']

  # The tasks that Updatebot can run. Only one of each task is currently permitted
  # optional
  tasks:
    - type: commit-alert
      branch: upstream-branch-name
      cc: ["bugzilla@email.address", "another@example.com"]
      needinfo: ["bugzilla@email.address", "another@example.com"]
      enabled: True
      filter: security
      frequency: every
      platform: windows
      blocking: 1234
    - type: vendoring
      branch: master
      enabled: False

      # frequency can be 'every', 'release', 'N weeks', 'N commits'
      # or 'N weeks, M commits' requiring satisfying both constraints.
      frequency: 2 weeks

Common Vendoring Operations

Update to the latest upstream revision:

./mach vendor /path/to/moz.yaml

Check for latest revision, returning no output if it is up-to-date, and a version identifier if it needs to be updated:

./mach vendor /path/to/moz.yaml --check-for-update

Vendor a specific revision:

./mach vendor /path/to/moz.yaml -r $REVISION --force

In the presence of patches, two steps are needed:

1. Vendor without applying patches (patches are applied after update-actions) through --patch-mode none

2. Apply patches on updated sources through --patch-mode only

In the absence of patches, a single step is needed, and no extra argument is required.

Vendoring Actions

Vendoring actions in the moz.yaml file can be configured to run either before or after patches are applied using separate sections:

• Actions in update-actions run before patches are applied

• Actions in post-patch-actions run after patches are applied

This separation is useful when you need to run scripts that depend on Mozilla-specific patches being applied first, such as:

• Code generation scripts that need patched configuration files

• Build system updates that depend on patched build definitions

• Processing steps that require Mozilla-specific modifications to be in place

Example:

# Actions that run before patches are applied
update-actions:
  - action: run-script
    script: '{yaml_dir}/pre_patch_script.sh'
    cwd: '{yaml_dir}'

# Actions that run after patches are applied
post-patch-actions:
  - action: run-script
    script: '{yaml_dir}/post_patch_script.sh'
    cwd: '{yaml_dir}'
    args: ['{revision}']