Handling Large Files in Git: LFS and Alternatives
Git is bad at large binaries. Learn why, how Git LFS works, and how to clean a repository that already has them.
By Velloreth
Why Git struggles
Git stores a full copy of each version of a file. For text, this is efficient, because Git compresses well and similar versions share data. For binaries, it is not: a compressed image or a video does not compress further, and each version is stored in full.
A 100MB file that changes ten times adds roughly a gigabyte to the repository. And it stays there forever, because the history is immutable.
The consequences:
- Slow clones. Everyone downloads the full history.
- Slow operations.
git log,git blame, andgit statusall get slower. - Hosting limits. GitHub warns above 1GB and blocks above 5GB.
Finding the culprits
List the largest objects:
git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| awk '/^blob/ {print $3, $4}' \
| sort -rn \
| head -20
This prints the twenty largest blobs with their paths.
Use git-sizer:
git-sizer --verbose
This reports the repository size, the largest blobs, and the largest trees, with a rating for each.
Check the pack size:
du -sh .git
Git LFS
Git LFS replaces large files with small text pointers and stores the content separately.
Install:
git lfs install
Track a file type:
git lfs track "*.psd"
git lfs track "*.mp4"
This creates .gitattributes, which is committed.
*.psd filter=lfs diff=lfs merge=lfs -text
*.mp4 filter=lfs diff=lfs merge=lfs -text
Commit as normal:
git add .gitattributes
git add design.psd
git commit -m "Add design file"
The repository stores a pointer:
version https://git-lfs.github.com/spec/v1
oid sha256:a1b2c3d4...
size 104857600
The actual content is uploaded to the LFS server.
Clone:
git clone https://github.com/user/repo.git
LFS files are downloaded on checkout. To skip them:
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/user/repo.git
Fetch LFS content later:
git lfs pull
See what is tracked:
git lfs ls-files
The cost of LFS
Storage quota. GitHub gives a limited amount of free LFS storage and bandwidth. Exceeding it requires a paid plan.
A separate server. LFS content is not in the repository, so a clone depends on the LFS server being available.
Not a fix for history. Adding LFS does not remove large files that are already in the history. You must migrate.
Migrating existing files to LFS
git lfs migrate import --include="*.psd,*.mp4" --everything
This rewrites history to replace the matching files with pointers. It is destructive, so coordinate with the team and force-push afterwards.
Check the result:
git lfs ls-files
du -sh .git
Removing large files from history
If a large file should not be in the repository at all, remove it from history.
Using git filter-repo (the modern tool):
git filter-repo --path path/to/large-file --invert-paths
Using the BFG (a simpler alternative):
java -jar bfg.jar --strip-blobs-bigger-than 100M repo.git
Both rewrite history, which means:
- Every commit hash changes.
- Everyone must re-clone or reset.
- Open pull requests may break.
- The remote must be force-pushed.
This is a disruptive operation. Announce it, and do it when the team is not mid-release.
Preventing the problem
Do not commit build output. Add dist/, build/, and *.min.js to .gitignore.
Do not commit dependencies. node_modules and vendor belong in a lockfile, not the repository.
Do not commit media. Large images, videos, and design files belong in an asset store or LFS.
Generate what can be generated. A compiled binary, a bundled asset, or a generated PDF should come from a build step, not a commit.
Use an artifact store. CI artifacts, a CDN, or a package registry are the right home for build output.
Set a size limit in CI. A check that fails when a commit adds a file over a threshold catches the problem at the source.
# A pre-commit check
max_size=5242880 # 5MB
for file in $(git diff --cached --name-only); do
size=$(git cat-file -s $(git hash-object "$file") 2>/dev/null || echo 0)
if [ "$size" -gt "$max_size" ]; then
echo "$file is $size bytes, over the limit"
exit 1
fi
done
Alternatives to LFS
LFS is not the only option, and it is not always the best one.
A build step. If the large file can be generated, generate it. A compiled binary, a bundled asset, or a generated PDF should come from the build, not from a commit. This is the cleanest solution, because the repository stays small and the artifact is reproducible.
An artifact store. CI artifacts, a package registry, or an object store are the right home for build output. The repository holds the source, and the artifact is fetched when needed.
A CDN. Large media that the application serves belongs on a CDN, not in the repository. The repository holds a URL, and the CDN holds the bytes.
A separate repository. A repository of assets, cloned only when needed, keeps the main repository small. This is a middle ground between LFS and an artifact store.
A package. A large binary that is a dependency can be published as a package and installed, which puts it in the package manager’s cache rather than the Git history.
The decision: if the file is source, it belongs in the repository. If it is generated, it belongs in a build step. If it is an asset the application serves, it belongs on a CDN. LFS is for the case where a large file genuinely is source and must be versioned.
Practical guidance
- Do not commit large binaries.
- Use Git LFS for files that must be versioned.
- Remember deleting a file does not shrink the repository.
- Use
git lfs migrateto convert existing files. - Use
filter-repoto remove files from history. - Coordinate before rewriting history.
- Add a size check in CI.
- Prefer a build step over committing generated files.
For the related guides, see gitignore-patterns-guide and git-submodules-vs-subtrees. For the broader context, git-bisect-guide and git-submodules-vs-subtrees.
Quotable summary
Git stores every version of every file, so a large binary that changes often bloats the repository permanently, because a file removed in a later commit is still in the history. Git LFS stores large files outside the repository and commits a small pointer instead, which keeps clones fast, at the cost of a storage quota and a dependency on the LFS server. Deleting a large file does not shrink the repository, so you must rewrite history with git filter-repo or the BFG, which changes every commit hash and requires everyone to re-clone. The best fix is to not commit large binaries at all — use a build step, an artifact store, or a CDN, and add a size check in CI to catch the problem at the source.