Skip to content
Velloreth
Developer Tools 6 min read

Handling Large Files in Git: LFS and Alternatives

Git is bad at large binaries. Learn why, how Git LFS works, and how to clean a repository that already has them.

By

Why Git struggles

Git stores a full copy of each version of a file. For text, this is efficient, because Git compresses well and similar versions share data. For binaries, it is not: a compressed image or a video does not compress further, and each version is stored in full.

A 100MB file that changes ten times adds roughly a gigabyte to the repository. And it stays there forever, because the history is immutable.

The consequences:

  • Slow clones. Everyone downloads the full history.
  • Slow operations. git log, git blame, and git status all get slower.
  • Hosting limits. GitHub warns above 1GB and blocks above 5GB.

Finding the culprits

List the largest objects:

git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '/^blob/ {print $3, $4}' \
  | sort -rn \
  | head -20

This prints the twenty largest blobs with their paths.

Use git-sizer:

git-sizer --verbose

This reports the repository size, the largest blobs, and the largest trees, with a rating for each.

Check the pack size:

du -sh .git

Git LFS

Git LFS replaces large files with small text pointers and stores the content separately.

Install:

git lfs install

Track a file type:

git lfs track "*.psd"
git lfs track "*.mp4"

This creates .gitattributes, which is committed.

*.psd filter=lfs diff=lfs merge=lfs -text
*.mp4 filter=lfs diff=lfs merge=lfs -text

Commit as normal:

git add .gitattributes
git add design.psd
git commit -m "Add design file"

The repository stores a pointer:

version https://git-lfs.github.com/spec/v1
oid sha256:a1b2c3d4...
size 104857600

The actual content is uploaded to the LFS server.

Clone:

git clone https://github.com/user/repo.git

LFS files are downloaded on checkout. To skip them:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/user/repo.git

Fetch LFS content later:

git lfs pull

See what is tracked:

git lfs ls-files

The cost of LFS

Storage quota. GitHub gives a limited amount of free LFS storage and bandwidth. Exceeding it requires a paid plan.

A separate server. LFS content is not in the repository, so a clone depends on the LFS server being available.

Not a fix for history. Adding LFS does not remove large files that are already in the history. You must migrate.

Migrating existing files to LFS

git lfs migrate import --include="*.psd,*.mp4" --everything

This rewrites history to replace the matching files with pointers. It is destructive, so coordinate with the team and force-push afterwards.

Check the result:

git lfs ls-files
du -sh .git

Removing large files from history

If a large file should not be in the repository at all, remove it from history.

Using git filter-repo (the modern tool):

git filter-repo --path path/to/large-file --invert-paths

Using the BFG (a simpler alternative):

java -jar bfg.jar --strip-blobs-bigger-than 100M repo.git

Both rewrite history, which means:

  • Every commit hash changes.
  • Everyone must re-clone or reset.
  • Open pull requests may break.
  • The remote must be force-pushed.

This is a disruptive operation. Announce it, and do it when the team is not mid-release.

Preventing the problem

Do not commit build output. Add dist/, build/, and *.min.js to .gitignore.

Do not commit dependencies. node_modules and vendor belong in a lockfile, not the repository.

Do not commit media. Large images, videos, and design files belong in an asset store or LFS.

Generate what can be generated. A compiled binary, a bundled asset, or a generated PDF should come from a build step, not a commit.

Use an artifact store. CI artifacts, a CDN, or a package registry are the right home for build output.

Set a size limit in CI. A check that fails when a commit adds a file over a threshold catches the problem at the source.

# A pre-commit check
max_size=5242880  # 5MB
for file in $(git diff --cached --name-only); do
  size=$(git cat-file -s $(git hash-object "$file") 2>/dev/null || echo 0)
  if [ "$size" -gt "$max_size" ]; then
    echo "$file is $size bytes, over the limit"
    exit 1
  fi
done

Alternatives to LFS

LFS is not the only option, and it is not always the best one.

A build step. If the large file can be generated, generate it. A compiled binary, a bundled asset, or a generated PDF should come from the build, not from a commit. This is the cleanest solution, because the repository stays small and the artifact is reproducible.

An artifact store. CI artifacts, a package registry, or an object store are the right home for build output. The repository holds the source, and the artifact is fetched when needed.

A CDN. Large media that the application serves belongs on a CDN, not in the repository. The repository holds a URL, and the CDN holds the bytes.

A separate repository. A repository of assets, cloned only when needed, keeps the main repository small. This is a middle ground between LFS and an artifact store.

A package. A large binary that is a dependency can be published as a package and installed, which puts it in the package manager’s cache rather than the Git history.

The decision: if the file is source, it belongs in the repository. If it is generated, it belongs in a build step. If it is an asset the application serves, it belongs on a CDN. LFS is for the case where a large file genuinely is source and must be versioned.

Practical guidance

  • Do not commit large binaries.
  • Use Git LFS for files that must be versioned.
  • Remember deleting a file does not shrink the repository.
  • Use git lfs migrate to convert existing files.
  • Use filter-repo to remove files from history.
  • Coordinate before rewriting history.
  • Add a size check in CI.
  • Prefer a build step over committing generated files.

For the related guides, see gitignore-patterns-guide and git-submodules-vs-subtrees. For the broader context, git-bisect-guide and git-submodules-vs-subtrees.

Quotable summary

Git stores every version of every file, so a large binary that changes often bloats the repository permanently, because a file removed in a later commit is still in the history. Git LFS stores large files outside the repository and commits a small pointer instead, which keeps clones fast, at the cost of a storage quota and a dependency on the LFS server. Deleting a large file does not shrink the repository, so you must rewrite history with git filter-repo or the BFG, which changes every commit hash and requires everyone to re-clone. The best fix is to not commit large binaries at all — use a build step, an artifact store, or a CDN, and add a size check in CI to catch the problem at the source.

Frequently asked questions

Why does Git struggle with large files?
Git stores a full copy of each version of a file, and it compresses text well but binaries poorly. A 100MB file that changes ten times adds roughly a gigabyte to the repository, and it stays in the history forever.
What is Git LFS?
Git Large File Storage. It replaces large files with small text pointers in the repository and stores the actual content on a separate server. Cloning fetches the pointers, and the content is downloaded on checkout.
Does deleting a large file shrink the repository?
No. The file remains in the history, so the repository stays large. You must rewrite history with filter-repo or the BFG to remove it.
How do I find the largest files in a repository?
Use git rev-list --objects --all piped through git cat-file --batch-check to list object sizes, or a tool like git-sizer. Both show which files are responsible for the size.
What is the difference between Git LFS and a build step?
Git LFS keeps the file in version control with a pointer. A build step generates the file from source, so it is never committed. The build step is better when the file can be generated.
Can I use Git LFS with GitHub?
Yes, GitHub supports Git LFS with a storage quota. Exceeding the quota requires a paid plan, and the bandwidth is also metered.