easy-git-annex

The easy-git-annex package is a JavaScript/TypeScript API for git-annex and Git commands. git-annex is an integrated alternative to storing large files in Git. Git commands can be used without installing git-annex.

The easy-git-annex API is a wrapper over git-annex and Git commands. Applications pass JavaScript objects with command options and parameters to easy-git-annex which generates the appropriate command line and runs the command.

Each command is run asynchronously without blocking the Node.js event loop. The Promise returned from every command has the same structure and includes the command and its arguments, repository path, exit code, stdout, and stderr. All Promise behavior, including chaining and concurrency, can be used normally.

Helper functions assist your application with parsing command responses. Additional methods, such as getRepositories and getStatusGit, return JavaScript objects for tasks common to many applications.

Callbacks for stdout and stderr are available to show progress of time-consuming commands. Environment variables can also be specified.

Installation

Git must be installed and verified to run from the command line.

If you want to use git-annex, download, install, and verify it runs from the command line.

Installation of easy-git-annex can be performed with the command

npm install easy-git-annex

GitHub repositories can use the jstritch/setup-git-annex action to install git-annex in workflows. Workflows using jstritch/setup-git-annex can test git-annex applications on Linux, macOS, and Windows.

Usage

The easy-git-annex package is an ECMAScript module. Your code can reference the library statically

import * as anx from 'easy-git-annex';

or by using a dynamic import.

const anx = await import('easy-git-annex');

Obtain an accessor to use the GitAnnexAPI interface with the desired repository. The directory passed to createAccessor must exist but may be empty.

const myAnx = anx.createAccessor(repositoryPath);

An application may hold multiple accessors simultaneously.

Commands

Git and git-annex commands are exposed by methods on the GitAnnexAPI interface. A process is spawned to run each command asynchronously. The ApiOptions parameter, described below, can be used to influence process creation.

Command options

Many git-annex and Git commands accept one or more options. The command-line syntax for passing option values varies per command and option. Applications can choose to let easy-git-annex deal with the syntax by passing options in an object. To meet atypical requirements, an application can construct the command-line syntax options in a string array.

Applications passing options as an object have a consistent way to specify values since easy-git-annex inserts the correct delimiters. When constructing a command option object, key names containing hyphens must be enclosed in single or double quotation marks to be valid JavaScript identifiers.

Scalar values are passed normally: { '--jobs': 2 }.
To pass a flag that does not accept a value, supply null as the value: { '--verbose': null }.
A tuple of [string, string] is used for options requiring a key-value pair: { '--set': ['annex.largefiles', 'mimeencoding=binary'] }.
Arrays are used for options accepting more than one value and options which can be repeated: { '-e': ['foo', 'bar', 'baz'] }.

Any keys in the object not defined for the command are ignored by easy-git-annex.

Command file name parameters

Some commands accept relative paths of file names. Git and git-annex commands use forward slash path separators regardless of platform. The gitPath and gitPaths functions perform the conversions when necessary.

The gitPath and gitPaths functions are called internally by easy-git-annex for implemented relative path parameters. When building a low-level command or constructing an options string array, calling gitPath and gitPaths is the application's responsibility. The following JavaScript illustrates a low-level call of the Git add command

const rslt = await myAnx.runGit(['add', anx.gitPath(relativePath)]);

which is equivalent to

const rslt = await myAnx.addGit(relativePath);

Command API options

An application can control the environment variables passed to the command and register callbacks for stdout and stderr using the ApiOptions parameter. The apiOptions parameter is accepted by easy-git-annex command methods.

The fragment below clones the current environment and adds variable GIT_TRACE to the copy.

const anxEnv = Object.assign({}, process.env);
anxEnv['GIT_TRACE'] = '2';
const apiOptions = { env: anxEnv };

The JavaScript bind function can be used to pass this and other parameters to a callback as shown in the fragment below.

const apiOptions = { outHandler: this.onAnnexOut.bind(this) };

The use of callbacks is shown in the Examples section, below.

Command result

The Promise returned by every command contains uninterpreted information about the process in the CommandResult interface.

Any method is capable of throwing an Error. Any command that doesn't throw can return an exitCode indicating failure. Application design must account for these eventualities.

Documentation

All features are explained in the API documentation. Links to commonly used methods appear below.

Setup methods

clone Clones a repository into an empty directory.
describeAnx Changes the description of a repository.
initAnx Initializes a repository for use with git-annex.
initGit Creates an empty Git repository or reinitializes an existing one.
reinit Initializes a repository for use with git-annex specifying the uuid.
submodule Manages submodules.
uninit Stops the use of git-annex in a repository.

Configuration Methods

configAnx Gets or set git-annex configuration settings.
configGit Gets or set Git configuration settings.
group Gets or sets the group association of a repository.
groupwanted Gets or sets the groupwanted expression of a repository group.
mincopies Gets or sets the minimum number of copies.
numcopies Gets or sets the desired number of copies.
required Gets or sets the required content expression of a repository.
ungroup Removes a repository from a group previously set by the group command.
wanted Gets or sets the wanted content expression of a repository.

Contents Methods

addAnx Adds files to Git and git-annex.
addGit Adds file contents to the index.
addunused Adds back unused files.
clean Removes untracked files from the working tree.
commit Records changes to the repository.
copy Copies file content to or from another repository.
diff Shows changes between commits, commit and working tree, etc.
drop Removes file content from a repository.
dropunused Drops unused file content.
export Exports a tree of files to a special remote.
find Lists available files.
get Makes content of annexed files available.
getFileNames Obtains an array of filenames in the index and the working tree.
getStatusAnx Obtains an array describing the working tree status.
getStatusGit Obtains an array describing the working tree status.
grep Finds lines matching a pattern.
import Imports a tree of files from a special remote.
lock Locks files to prevent modification.
lsFiles Shows information about files in the index and the working tree.
move Moves file content to or from another repository.
oldkeys Lists keys used for old versions of annexed files.
mv Moves or renames a file, a directory, or a symlink.
rm Removes file content from the repository.
statusAnx Shows the working tree status.
statusGit Shows the working tree status.
unannex Undoes a git-annex add command.
unlock Unlocks files for modification.
unused Looks for unused file content.
whereis Lists repositories containing files.
whereused Finds which files use or used a key.

Branching methods

adjust Enters an adjusted branch.
branch Manages branches.
checkout Switches branches or restores working tree files.
cherryPick Applies changes introduced by existing commits.
getBranchName Obtains the current branch name.
getBranchNames Obtains an array of branch names.
getTagNames Obtains an array of tag names.
log Shows commit log entries.
mergeAnx Joins two or more development histories.
mergeGit Joins two or more development histories.
rebase Reapplies commits on top of another base tip.
reset Resets the current HEAD to the specified state.
restore Restores working tree files.
revert Reverts existing commits.
stash Saves the changes in a dirty working directory.
switch Shows changes between commits, commit and working tree, etc.
tag Creates, deletes, or lists tag objects.

Remotes methods

assist Adds files and synchronizes changes with remotes.
configremote Changes special remote configuration.
dead Hides a lost repository.
enableremote Enables use of an existing remote in the current repository.
expire Expires inactive repositories.
fetch Downloads objects and refs from another repository.
getRemoteNames Obtains an array of remote names.
initremote Creates a special remote.
pullAnx Pulls content from remotes.
pull Fetches from and integrates with another repository or a local branch.
pushAnx Pushes content to remotes.
push Updates remote refs along with associated objects.
remote Manages the set of tracked repositories.
renameremote Changes the name of a special remote.
satisfy Transfers and drops content as configured.
semitrust Sets a repository to the default trust level.
sync Synchronizes the local repository with remotes.
untrust Records that a repository is not trusted and could lose content at any time.

Inspection methods

forEachRef Reports information about each ref.
getBackends Obtains an array of key-value backends.
getBuildFlags Obtains an array of the git-annex build flags.
getRepositories Obtains an array identifying the current repositories.
getRepositoryInfo Obtains an object identifying the current repository.
getSpecialRemoteTypes Obtains an array of special remote types.
info Obtains information about an item or the repository.
list Shows which remotes contain files.
revParse Picks out and massages parameters.
show Displays various types of objects.

Generic functions

getBranches Returns information about Git branches in application-defined JavaScript objects.
getFinds Returns information about available files in application-defined JavaScript objects.
getLogs Returns information about Git commits in application-defined JavaScript objects.
getRefs Returns information about Git refs in application-defined JavaScript objects.
getShows Returns information about Git objects in application-defined JavaScript objects.
getTags Returns information about Git tags in application-defined JavaScript objects.
getWhereis Returns information about repositories containing files in application-defined JavaScript objects.
listFiles Returns information about index and working tree files in application-defined JavaScript objects.

Maintenance Methods

forget Prunes git-annex branch history.
fsckAnx Verifies the validity of objects in git-annex.
fsckGit Verifies the connectivity and validity of objects in Git.
repair Recovers a broken Git repository.

Version methods

getVersionAnx Obtains build information about the local git-annex installation in a JavaScript object.
getVersionGit Obtains build information about the local Git installation in a JavaScript object.
versionAnx Obtains build information about the local git-annex installation.
versionGit Obtains build information about the local Git installation.

Low-level methods

runAnx Runs any git-annex command.
runGit Runs any Git command.

Contributions

If you would like to suggest a new feature or report a problem, please create an issue.

If you would like to improve the easy-git-annex code, please read CONTRIBUTING.md.

I am an independent developer. If you find easy-git-annex helpful, please consider donating via Ko-fi, Liberapay, Patreon, or GitHub.

Examples

Create a repository

This example illustrates one way to create a git-annex repository and add some files. The example may be copied and run on your machine. When you invoke runExampleClick from a test application, it creates a temporary directory, prepares the directory for Git and git-annex, adds one large and one small file to a subdirectory, and reports success or failure.

The repository remains on your system for study. Once the repository exists, you can

List annexed files using the git annex list command.
Add remote repositories using git clone followed by git annex init.
Add special remotes with git annex initremote. The directory remote type is great for learning.
Save new files, deleted files, and file modifications using the commands git annex add followed by git commit.
Use the git annex push and git annex pull commands to copy files between a repository and a special remote or another repository.
Experiment with other commands.

import * as anx from 'easy-git-annex';
import * as os from 'os';
import * as path from 'path';
import { promises as fs } from 'fs';

async function setupAnnexClient(repositoryPath: string, description: string, largefiles: string): Promise<void> {
  const myAnx = anx.createAccessor(repositoryPath);

  let rslt = await myAnx.initGit();
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.configGit({ set: ['push.followTags', 'true'] });
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.configGit({ set: ['receive.denyCurrentBranch', 'updateInstead'] });
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.initAnx(description);
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.wanted('here', 'standard');
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.group('here', 'manual');
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.configAnx({ '--set': ['annex.largefiles', largefiles] });
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }
}

async function addFiles(repositoryPath: string, relativePaths: string | string[], commitMessage: string): Promise<void> {
  const myAnx = anx.createAccessor(repositoryPath);

  let rslt = await myAnx.addAnx(relativePaths);
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }

  rslt = await myAnx.commit(relativePaths, { '--message': commitMessage });
  if (rslt.exitCode !== 0) { throw new Error(rslt.toCommandResultString()); }
}

export async function runExampleClick(): Promise<void> {
  try {
    // create a directory
    const repositoryPath = await fs.mkdtemp(path.join(os.tmpdir(), 'anx-client-'));

    // setup a git-annex client
    const largefiles = 'include=*.xtx or include=*.jpg';
    await setupAnnexClient(repositoryPath, 'images', largefiles);

    // add a subdirectory
    const exampleDir = 'january';
    await fs.mkdir(path.join(repositoryPath, exampleDir));

    // add a small and large file
    const smallFile = path.join(exampleDir, 'small file.txt');
    const largeFile = path.join(exampleDir, 'large file.xtx');
    await fs.writeFile(path.join(repositoryPath, smallFile), 'small file stored in Git\n');
    await fs.writeFile(path.join(repositoryPath, largeFile), 'large file stored in git-annex\n');
    await addFiles(repositoryPath, [smallFile, largeFile], 'add two files');

    console.log(`Created ${repositoryPath}`);
  } catch (e: unknown) {
    console.error(e);
  }
}

Show progress of long-running commands

To make the addFiles function, above, display progress as files are added to git-annex, add the reportProgress function shown below. The safeParseToArray function converts the received string to an array of JavaScript objects meeting the ActionProgress interface. Each object is then written to console.info.

function reportProgress(data: string): void {
  const progress = anx.safeParseToArray(anx.isActionProgress, data);
  progress.forEach((e) => { console.info(`${e['percent-progress']} ${e.action.command} ${e.action.file} ${e['byte-progress'].toLocaleString()} / ${e['total-size'].toLocaleString()}`); });
}

Then modify the addAnx line in addFiles to include the AddAnxOptions and ApiOptions parameters as shown below. The --json-progress option requests JSON progress be written to stdout as files are added. The outHandler establishes the reportProgress function to be called as information becomes available on stdout.

  let rslt = await myAnx.addAnx(relativePaths, { '--json-progress': null }, { outHandler: reportProgress });

When the example is run, progress messages appear on the console for each annexed file before the command completes.

Get application-defined JavaScript objects

Several generic functions are included in easy-git-annex. These functions return JavaScript objects defined by your application. This example uses the generic function getTags to obtain tag objects. The pattern is similar for the other generic functions.

First declare the interface your application requires.

export interface FooTag {
  name: string;
  objectName: string;
  taggerDate?: Date;
  contents?: string;
}

Then write a type predicate to determine if an object meets the interface requirements.

export function isFooTag(o: unknown): o is FooTag {
  if (!anx.isRecord(o)) { return false; }
  if (!anx.isString(o['name'])) { return false; }
  if (!anx.isString(o['objectName'])) { return false; }
  if ('taggerDate' in o && !anx.isDate(o['taggerDate'])) { return false; }
  if ('contents' in o && !anx.isString(o['contents'])) { return false; }
  return true;
}

Write a function to return your tag objects. The getFooTags function takes it's caller's arguments, sets up a getTags call, and returns the result.

export async function getFooTags(repositoryPath: string, tagName?: string, ignoreCase?: boolean): Promise<FooTag[]> {

  const columns: [string, anx.Parser?][] = [
    ['name'],
    ['objectName'],
    ['taggerDate', anx.parseUnixDate],
    ['contents', anx.parseOptionalString],
  ];

  const options: anx.ForEachRefOptions = {
    '--format': '%(refname:lstrip=2)%09%(objectname)%09%(taggerdate:unix)%09%(contents:lines=1)',
    '--sort': ['*refname'],
    ...ignoreCase === true && { '--ignore-case': null }
  };

  return anx.getTags(isFooTag, columns, repositoryPath, options, tagName);
}

Your application can make the following call to get a list of all tags beginning with the letter v:

const fooTags = await getFooTags(repositoryPath, 'v*');

The fooTags array can then be used in your application normally.

easy-git-annex

easy-git-annex

Installation

Usage

Commands

Command options

Command file name parameters

Command API options

Command result

Documentation

Setup methods

Configuration Methods

Contents Methods

Branching methods

Remotes methods

Inspection methods

Generic functions

Maintenance Methods

Version methods

Low-level methods

Contributions

Examples

Create a repository

Show progress of long-running commands

Get application-defined JavaScript objects

Settings

Member Visibility

Theme

On This Page