The recent blog post Commits are snapshots, not diffs did a good job explaining away a common git misconception, and helped me finally understand it. To really wrap my head around it, I checked it empirically.

The misconception starts because git presents commits as diffs, and lets you manipulate them (rebase, cherry-pick, etc) as if they were diffs. But internally, git commits are not diffs, they are complete copies of the file at each revision that changes the file.

At first glance, this seems dumb: why store the whole file over again just because one line changed? The reason is speed and immutability. If git stored each commit as a diff against the previous version (as RCS did), then getting the latest version of a file would require replaying all the diffs against the very first version of the file, getting slower and slower as the repo accumulated more commits. This means the most common checkout would get worse and worse over time.

If git stored the latest version of a file, and diffs going backward in time (as Subversion does), then getting older versions would get slower and slower, which isn’t so bad. But it would require re-writing the previous commit each time a new commit was made. This would ruin git’s hash-based immutability.

So, surprisingly, git stores the full contents of the file each time the file changes. I wanted to see this for myself, so I did an experiment.

First, make a new git repo:

$ mkdir gitsize
$ cd gitsize
$ git init
Initialized empty Git repository in /tmp/gitsize/.git/

I used a Python program (makebig.py) to create large files with repeatably random contents and one changeable word in the middle:

# Make a 1Mb randomish file, repeatably

import random, sys

random.seed(int(sys.argv[1]))

for lineno in range(2048):
    if lineno == 1000:
        print(sys.argv[2])
    print("".join(chr(random.randint(32, 126)) for _ in range(512)))

Let’s make a big file with “one” in the middle, and commit it:

$ python /tmp/makebig.py 1 one > one.txt
$ ls -lh
total 2136
-rw-r--r--  1 ned  wheel   1.0M Dec 19 11:13 one.txt
$ git add one.txt
$ git commit -m "One"
[master (root-commit) 8fceff3] One
 1 file changed, 2049 insertions(+)
 create mode 100644 one.txt

Git stores everything in the .git directory, with the file contents in the .git/objects directory:

$ ls -Rlh .git/objects/*
.git/objects/13:
total 1720
-r--r--r--  1 ned  wheel   859K Dec 19 11:14 b581d8695866f880eac2fef47c2594bbebbb3b

.git/objects/7d:
total 8
-r--r--r--  1 ned  wheel    52B Dec 19 11:14 32a67a911e8a04ad1703712481afe93b00c7af

.git/objects/8f:
total 8
-r--r--r--  1 ned  wheel   127B Dec 19 11:14 ceff3e3926764197742b01639a42765e34cd72

.git/objects/info:

.git/objects/pack:

Git stores three kinds of things: blobs, trees, and commits. We now have one of each. Blobs store the file contents. You can see the b581d8 blob is 859Kb, which is our 1Mb file with a little compression applied.

Now we change the file just a little bit by writing it over again with a different word in the middle:

$ python /tmp/makebig.py 1 one-changed > one.txt
$ git diff
diff --git a/one.txt b/one.txt
index 13b581d..b13026a 100644
--- a/one.txt
+++ b/one.txt
@@ -998,7 +998,7 @@ wLh&#DvF%em\Bb}^Y<gk?5vR8npq{ ~".][T|@.At@~fGYf<0/=cth`e}/}='qBFb&FP?+ENmAA:_g+0
 u$d|\v=y$oi@\, (o`=a49|!r\LL^B:y.f)*@5^bR\,Ck=i (.. snipped)
 lbY#m++>32X>^gh\/q34})uxZ"e/p;Ybb9\k,UTLPb*?3l7 (.. snipped)
 B11\\!x]jM9m't"KD%|,&r(lfh%vfT}~{jOQYBb?|TZ(<<R (.. snipped)
-one
+one-changed
 >Mu2P-/=8Z+A&"#@'"8*~fb]kkn;>}Ie.)wGjjHsbO5Nw]" (.. snipped)
 Vl {Q)k|{E!vF*@S')U5bK3u1fInN*ZrIe{-qXW}Fr`6*#N (.. snipped)
 3lF#jR!"JxXjAvih 4I6E\W:Y.*}b@eZ8xl-"*c/!pe"$Mx (.. snipped)

Commit the change, and we can look again at the .git storage:

$ git commit -am "One, changed"
[master a2410c8] One, changed
 1 file changed, 1 insertion(+), 1 deletion(-)
$ ls -Rlh .git/objects/*
.git/objects/0e:
total 8
-r--r--r--  1 ned  wheel    52B Dec 19 11:22 2de9f34b9140c3e99c5d5106a1078d22aa9063

.git/objects/13:
total 1720
-r--r--r--  1 ned  wheel   859K Dec 19 11:14 b581d8695866f880eac2fef47c2594bbebbb3b

.git/objects/7d:
total 8
-r--r--r--  1 ned  wheel    52B Dec 19 11:14 32a67a911e8a04ad1703712481afe93b00c7af

.git/objects/8f:
total 8
-r--r--r--  1 ned  wheel   127B Dec 19 11:14 ceff3e3926764197742b01639a42765e34cd72

.git/objects/a2:
total 8
-r--r--r--  1 ned  wheel   163B Dec 19 11:22 410c8b799b7829e1360649011754739e0a5c50

.git/objects/b1:
total 1720
-r--r--r--  1 ned  wheel   859K Dec 19 11:22 3026a4c10928821aa2b89b3e67d766dfbd533a

.git/objects/info:

.git/objects/pack:

Now, as promised, there are two blobs, each 859Kb. The original file contents are still in blob b581d8, and there’s a new blob (3026a4) to hold the updated contents.

Even though we changed just one line in a 2000-line file, git stores the full contents of both revisions of the file.

Isn’t this terrible!? Won’t my repos balloon to unmanageable sizes? Nope, because git has another trick up its sleeve. It can store those blobs in “pack files”, which store repeated sequences of bytes once.

Git will automatically pack blobs when it makes sense to, but we can ask it to explicitly in order to see them in action:

$ git gc --aggressive
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), done.
Total 6 (delta 1), reused 0 (delta 0), pack-reused 0
$ ls -Rlh .git/objects/*
.git/objects/info:
total 16
-rw-r--r--  1 ned  wheel   1.2K Dec 19 11:41 commit-graph
-rw-r--r--  1 ned  wheel    54B Dec 19 11:41 packs

.git/objects/pack:
total 1720
-r--r--r--  1 ned  wheel   1.2K Dec 19 11:41 pack-a0d87c64abc0f03070fd14449891fe20ca98926b.idx
-r--r--r--  1 ned  wheel   855K Dec 19 11:41 pack-a0d87c64abc0f03070fd14449891fe20ca98926b.pack

Now instead of individual blob files, we have one pack file. And it’s a little smaller than either of the blobs!

This may seem like a semantic game: doesn’t this show that commits are deltas? It’s not the same, for a few reasons:

• Reconstructing a file doesn’t require revisiting its history. Every revision is available with the same amount of effort.
• The sharing between blobs is at a conceptually different layer than the blob storage. Git stores a commit as a full snapshot of all of the files’ contents. The file contents might be stored in a shared-bytes way within the pack files.
• The object model is full-file contents in blobs, and commits referencing those blobs. If you removed pack files from the implementation, the conceptual model and all operations would work the same, just take more disk space.
• The storage savings in a pack file are not limited to a single file. If two files (or two revisions of two different files) are very similar, their bytes will be shared.

To demonstrate this last point, we’ll make another file with almost the same contents as one.txt:

$ python /tmp/makebig.py 1 two > two.txt
$ ls -lh
total 4280
-rw-r--r--  1 ned  wheel   1.0M Dec 19 11:18 one.txt
-rw-r--r--  1 ned  wheel   1.0M Dec 19 11:49 two.txt
$ git add two.txt
$ git commit -m "Two"
[master 079baa5] Two
 1 file changed, 2049 insertions(+)
 create mode 100644 two.txt
$ git gc --aggressive
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), done.
Total 9 (delta 2), reused 4 (delta 0), pack-reused 0
$ ls -Rlh .git/objects/*
.git/objects/info:
total 16
-rw-r--r--  1 ned  wheel   1.2K Dec 19 11:50 commit-graph
-rw-r--r--  1 ned  wheel    54B Dec 19 11:50 packs

.git/objects/pack:
total 1720
-r--r--r--  1 ned  wheel   1.3K Dec 19 11:50 pack-36b681bfc8ebef963bb8a7dcfe65addab822f5d4.idx
-r--r--r--  1 ned  wheel   855K Dec 19 11:50 pack-36b681bfc8ebef963bb8a7dcfe65addab822f5d4.pack

Now we have two separate source files in our working tree, each 1Mb. But in the .git storage there is still just one 855Kb pack file. The parts of one.txt and two.txt that are the same are only stored once.

As another example, let’s change two.txt completely by using a different random seed, commit it, then change it back again:

$ python /tmp/makebig.py 2 two > two.txt
$ git commit -am "Two, completely changed"
[master 6dac887] Two, completely changed
 1 file changed, 2049 insertions(+), 2049 deletions(-)
 rewrite two.txt (86%)
$ python /tmp/makebig.py 1 two > two.txt
$ git commit -am "Never mind, I liked it the old way"
[master c06ad2f] Never mind, I liked it the old way
 1 file changed, 2049 insertions(+), 2049 deletions(-)
 rewrite two.txt (86%)

Looking at the storage, our pack file is twice the size, because we’ve had two completely different 1Mb-chunks of data. But thinking about two.txt, its first and third revisions were nearly identical, so they can share bytes in the pack file:

$ git gc --aggressive
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (11/11), done.
Writing objects: 100% (13/13), done.
Total 13 (delta 3), reused 6 (delta 0), pack-reused 0
$ ls -Rlh .git/objects/*
.git/objects/info:
total 16
-rw-r--r--  1 ned  wheel   1.3K Dec 19 11:58 commit-graph
-rw-r--r--  1 ned  wheel    54B Dec 19 11:58 packs

.git/objects/pack:
total 3432
-r--r--r--  1 ned  wheel   1.4K Dec 19 11:58 pack-49f264f911dc97e529dc56a4f6ad450f8013f720.idx
-r--r--r--  1 ned  wheel   1.7M Dec 19 11:58 pack-49f264f911dc97e529dc56a4f6ad450f8013f720.pack

If git stored diffs, we’d need two different megabyte-sized diffs for the two complete changes we’ve made to two.txt.

Note that in this experiment I have used “git gc” to force the storage into its most compact form. You typically wouldn’t do this. Git will automatically repack files when it makes sense to.

Git doesn’t store diffs, it stores the complete contents of the file at each revision. But beneath those full-file snapshots is clever redundancy-removing byte storage that makes the total size even smaller than a diff-based system could achieve.

If you want to know more, How Git Works is a good overview, and Git Internals - Git Objects is the authoritative reference.