# How Git Actually Works: Objects, Refs, the Index, and Packfiles > A ground-up explanation of Git's internals, faithful to the official documentation: the content-addressed object database, the 41-byte ref, reftable, the DIRC index and conflict stages 1/2/3, three-way merges under the ort strategy, packfile delta compression, and the SHA-256 and Git 3.0 breaking changes — with plumbing-command output you can reproduce yourself. - Published: 2026-08-16 - Author: 友田 陽大 - Tags: Git, バージョン管理, アーキテクチャ設計, 信頼性, セキュリティ - URL: https://tomodahinata.com/en/blog/git-internals-object-model-refs-index-packfile-guide - Category: Git internals & repository operations ## Key points - Git stores snapshots, not diffs. An object's name is the hash of 'header + content', so identical file content converges to a single object no matter how many times you commit it - A branch is a 41-byte text file. That is why creating one is instant, and why rebase and amend are not edits but the creation of entirely new objects - A conflict is three generations living in the index at once — stage 1 (common ancestor), 2 (ours), 3 (theirs). You can always see them with git ls-files -u - The default merge strategy, ort, handles multiple common ancestors by merging the ancestors into a virtual tree and using that as the three-way merge base (stated in the official docs) - Git 3.0 will change the default hash to SHA-256, the default ref storage to reftable, the default branch name to main, and safe.bareRepository to explicit — all documented in the official BreakingChanges file --- You ran `git reset --hard` and three hours of work vanished. You ran `git push --force` and wiped out a teammate's commits. A merge produced obviously wrong output and you cannot explain where it went wrong. Three different accidents, one root cause: **you are memorizing commands without knowing which of Git's three data structures — objects, refs, and the index — each one touches.** Git's command surface is terrible to memorize. `checkout` used to do two unrelated jobs, "switch branches" and "restore files" (which is why v2.23 split it into `switch` and `restore`). `reset` changes how many of the three trees it moves based on a flag. The porcelain is a pile of historical accident. **The internals, by contrast, are startlingly simple.** Git is a key-value store addressed by content hash, a set of 41-byte pointers into it, and a draft of the next commit. Once that clicks, commands stop being things you memorize and become things you **derive**. This article takes Git's official documentation (as of Git 2.55) and the `Documentation/` tree of the `git/git` repository as primary sources, and takes the internals apart. Every command output below was captured on **git version 2.50.1**. By the end you will be able to say, out loud, which tree an accident moved. --- ## 1. Git is a content-addressed key-value store ### Experiment: you can compute the hash yourself An object name (what people call the commit ID or SHA) is neither random nor sequential. **It is computed deterministically from the content.** ```bash $ echo -n "what is up, doc?" | git hash-object --stdin bd9dbf5aae1a3862dd1526723246b20206e5fc37 ``` You can reproduce that value without Git. Git does not hash the content directly — it **prepends a header** of the form ` \0` first. ```bash $ printf 'blob 16\0what is up, doc?' | shasum bd9dbf5aae1a3862dd1526723246b20206e5fc37 - ``` Identical. `hash-object` is nothing more than "prepend a header, take SHA-1." Why the header? **To put the type into the namespace.** The same byte sequence stored as a `blob` and as a `tree` gets different object names. Type confusion is designed out. ### Experiment: what it looks like on disk With `-w`, the object is written into the object database. ```bash $ printf 'hello\n' > a.txt $ git hash-object -w a.txt ce013625030ba8dba906f756967f9e9ca394464a $ find .git/objects -type f .git/objects/ce/013625030ba8dba906f756967f9e9ca394464a ``` The path splits the name: **first two characters become the directory, the remaining 38 the filename.** This keeps hundreds of thousands of files out of one directory, where many filesystems degrade sharply. The content is zlib-compressed. Inflate it and the header is right there. ```python import zlib data = open('.git/objects/ce/013625030ba8dba906f756967f9e9ca394464a', 'rb').read() print(repr(zlib.decompress(data))) # => b'blob 6\x00hello\n' ``` `blob 6\0hello\n` — exactly the bytes that were hashed, stored verbatim. Git's object store is, in this sense, **remarkably literal**. ### Why this matters This is where "Git stores snapshots, not diffs" comes from. Every commit points at a complete tree, but **identical content produces an identical object name**, so unchanged files never create new objects. A commit that touches one file out of ten thousand adds one blob and rebuilds only the trees along that path. And "same content always has the same name" *is* an integrity check. Inflate a downloaded object, re-hash it, and both tampering and transfer corruption show up. That is what `git fsck` does. --- ## 2. The four object types Git has exactly four. | Type | Role | Contents | |---|---|---| | `blob` | File content | Raw bytes (no filename, no permissions) | | `tree` | Directory | A list of "mode, type, object name, name" | | `commit` | Snapshot plus metadata | Root tree, parent commits, author/committer, message | | `tag` | Annotated tag | Target object, tagger, message, optional signature | ### tree: this is where filenames live Blobs have no name. Names and permissions belong to the `tree`. ```bash $ git cat-file -p 2e81171448eb9f2ee3821e3d447aa6b2fe3ddba1 100644 blob ce013625030ba8dba906f756967f9e9ca394464a a.txt ``` The mode is not a Unix permission bitmask; it is a **small, fixed vocabulary**. | Mode | Meaning | |---|---| | `100644` | Regular file | | `100755` | Executable file | | `120000` | Symbolic link | | `040000` | Subdirectory (tree) | | `160000` | gitlink (a submodule's commit reference) | `gitformat-index(5)` states that for the index's mode field, "only 0755 and 0644 are valid for regular files." **Git records nothing about permissions beyond the executable bit.** That is why `chmod 600` is not version-controlled — and why you must never rely on Git to protect the permissions of a sensitive file. ### commit: what makes the ID change ```bash $ git cat-file -p 8994a20d11cbdb935ae7a857349e19a815e2e61e tree 2e81171448eb9f2ee3821e3d447aa6b2fe3ddba1 author Demo 1786806000 +0900 committer Demo 1786806000 +0900 first ``` That is the entire commit object. A merge commit lists several `parent` lines; a signed commit carries a `gpgsig` line. What matters is that **the hash of this whole text is the commit ID**. Three consequences follow directly. 1. **`git commit --amend` does not edit a commit.** A different message means different bytes means a different ID. The original stays exactly where it was; only the ref moves. 2. **Rebase does not move history, it recreates it.** Change a parent and the child's ID changes, and so does its child's, all the way down. That is why rebasing a shared branch is incompatible with everyone else's history. 3. **A signature is part of the commit.** `gpgsig` lives inside the commit object, so the commit ID commits to the signature. You cannot swap a signature in afterwards. ### Demo: building a commit out of plumbing alone What the porcelain does internally is obvious once you reproduce it with plumbing. ```bash $ git init -q -b main demo && cd demo $ git config user.email a@example.com && git config user.name Demo # 1. Put content into the object database (this creates a blob) $ printf 'hello\n' > a.txt $ git hash-object -w a.txt ce013625030ba8dba906f756967f9e9ca394464a # 2. Register it in the index (this is what git add does) $ git update-index --add --cacheinfo 100644,ce013625030ba8dba906f756967f9e9ca394464a,a.txt $ git ls-files --stage 100644 ce013625030ba8dba906f756967f9e9ca394464a 0 a.txt # 3. Turn the index into a tree object $ git write-tree 2e81171448eb9f2ee3821e3d447aa6b2fe3ddba1 # 4. Turn the tree into a commit object (no parent, so this is the root commit) $ git commit-tree 2e81171448eb9f2ee3821e3d447aa6b2fe3ddba1 -m "first" 8994a20d11cbdb935ae7a857349e19a815e2e61e # 5. Point the branch at that commit (only now is anything "committed") $ git update-ref refs/heads/main 8994a20d11cbdb935ae7a857349e19a815e2e61e ``` `git commit` is **steps 2 through 5 rolled into one command**. Conversely, if that final `update-ref` fails, you are left with orphaned objects. That is why "I committed but it isn't in the log" is a refs problem far more often than an objects problem. --- ## 3. Refs: a branch is 41 bytes ### A branch is a file ```bash $ cat .git/refs/heads/main 8994a20d11cbdb935ae7a857349e19a815e2e61e $ wc -c .git/refs/heads/main 41 .git/refs/heads/main ``` Forty hex characters plus a newline: **41 bytes**. That is a branch. Which explains: - **Creating a branch is instant.** You write one 41-byte file. - **Branches never make a repository heavier.** - **Deleting a branch does not delete history.** You removed a pointer; the objects are still there (which is why you can get them back). `HEAD` is slightly different — it is a **symbolic ref**. ```bash $ cat .git/HEAD ref: refs/heads/main ``` `HEAD` pointing at another ref is the normal state; `HEAD` holding an object name directly is a **detached HEAD**. The reason "commits made on a detached HEAD disappear" is that advancing `HEAD` updates no branch file, so the moment you switch branches nothing points at that commit any more (the reflog still does). ### packed-refs and reftable At tens of thousands of refs, "tens of thousands of 41-byte files" falls apart. The old answer is `.git/packed-refs` (all refs in one text file); Git's new answer is **reftable**. `git-init(1)` is unambiguous: > `--ref-format=` — the valid values are `files`, for loose files with packed-refs, which is the default, and `reftable`, for the reftable format. And the official `BreakingChanges.adoc` announces that Git 3.0 will make **reftable the default for new repositories**, listing the reasons. In summary, five of them: 1. **On case-insensitive filesystems (Windows/macOS) you cannot hold `refs/heads/Foo` and `refs/heads/foo` at once.** reftable does not encode ref names as filesystem paths, so the problem disappears. 2. **macOS normalizes Unicode in path names**, so two differently-encoded ref names cannot coexist. Same resolution. 3. **Deleting a ref with the `files` backend requires rewriting the whole `packed-refs` file** — dozens of megabytes, sometimes gigabytes, in large repositories. reftable uses tombstone markers, so no full rewrite. 4. **Writing multiple refs at once is not atomic with `files`.** Other processes can observe an in-between state while a transaction is being committed. 5. **Writing many refs at once is slow**, because each one becomes its own file. reftable outperforms `files` here "by multiple orders of magnitude," in the docs' own words. reftable is a **block-based binary format** with prefix compression on ref names, maintained as an **immutable stack** of tables listed in `.git/reftable/tables.list`. Each update appends a new table, and swapping `tables.list` is the atomic commit. Readers search the stack newest-first. Trying it is one command. ```bash $ git init --ref-format=reftable myrepo $ ls myrepo/.git/reftable/ 0x000000000001-0x000000000001-0d8df5e0.ref tables.list $ cat myrepo/.git/reftable/tables.list 0x000000000001-0x000000000001-0d8df5e0.ref ``` The filename is `${min_update_index}-${max_update_index}-${random}.ref`. Each update adds a table and appends a line to `tables.list`. Migrating an existing repository is `git refs migrate --ref-format=reftable` (check first that no mirror or hook reads ref paths directly). ### A remote-tracking branch is "what the remote looked like last time" `refs/remotes/origin/main` is **not the remote's current value**. It is a snapshot from your last `fetch`. Holding that straight is the key to judging `--force-with-lease` correctly in the next section. ### Applied: atomic ref updates in CI When ref updates must be transactional, use `git update-ref --stdin`. **Either all of them apply or none of them do.** ```bash # Update the release branch and the release tag indivisibly $ git update-ref --stdin <<'EOF' start update refs/heads/release ceb1a2f9c8ec5f4e2a1d0b3c6e7f8a9b0c1d2e3f create refs/tags/v1.4.0 ceb1a2f9c8ec5f4e2a1d0b3c6e7f8a9b0c1d2e3f prepare commit EOF ``` The third argument of `update []` is **the value you expect the ref to currently hold** — optimistic locking, on the server side, detecting concurrent CI jobs stepping on each other. Requirements that branch protection rules cannot express ("move several refs together") can only be met safely this way. --- ## 4. The index: the third tree ### What is inside `.git/index` What people call "the staging area" is a single binary file, `.git/index`. Per `gitformat-index(5)`: - A 12-byte header: the signature `{'D','I','R','C'}` (short for "dircache"), a version number (2, 3, and 4 are currently supported), and the number of entries - A run of sorted index entries - Extensions - A checksum over everything above An entry holds more than an object name and a path. **It holds the full result of `stat(2)`** — ctime (seconds and nanoseconds), mtime (seconds and nanoseconds), dev, ino, mode, uid, gid, and file size. That is where `git status` gets its speed. Git is not re-reading every file in the working tree; it **compares the result of `lstat()` against the index record and skips reading content when they match**, reading only what might have changed. So when `git status` is slow, the cause is almost always in this layer. In order of effectiveness: | Symptom | Setting that helps | What it does | |---|---|---| | Very many files | `git config core.untrackedCache true` | Per-directory cache for untracked-file discovery (the `UNTR` extension) | | Virtual or network filesystem | `git config core.fsmonitor true` | Uses OS change notifications to narrow the scan to what actually changed (the `FSMN` extension) | | Monorepo, narrow work area | cone-mode sparse-checkout | Collapses out-of-scope paths into directory-level entries (sparse index) | The sparse index is especially effective. `gitformat-index(5)` explains: > if sparse-checkout is enabled in cone mode (`core.sparseCheckoutCone` is enabled) and the `extensions.sparseIndex` extension is enabled, then the index may contain entries for directories outside of the sparse-checkout definition. These entries have mode `040000`, include the `SKIP_WORKTREE` bit, and the path ends in a directory separator. In other words, **a 100,000-file subtree collapses into a single index entry**. Since entry count *is* the cost of `git status`, this is not a linear improvement — it is an order-of-magnitude one. ### The cache tree extension: why `write-tree` is fast The index's `TREE` extension (the cache tree) **caches the already-computed tree object name for each directory**. Unchanged directories are immediately known not to need rebuilding, which is what keeps `git write-tree` — and therefore `git commit` — fast even in large repositories. ### What a conflict really is: stages 1, 2, and 3 Normally index entries are at **stage 0**. When a merge conflicts, **three generations occupy the same path**. ```bash $ git merge feature Auto-merging f.txt CONFLICT (content): Merge conflict in f.txt Automatic merge failed; fix conflicts and then commit the result. $ git ls-files -u 100644 83db48f84ec878fbfb30b46d16630e944e34f205 1 f.txt 100644 d791e9b8158e2be3a792fe1881d57828989a3449 2 f.txt 100644 00dbdcfd2b3a2c6a3a0facd9c753578741dc921e 3 f.txt ``` | Stage | Meaning | How to read it | |---|---|---| | 1 | Common ancestor (base) | `git show :1:f.txt` | | 2 | Our side (ours / HEAD) | `git show :2:f.txt` | | 3 | Their side (theirs / MERGE_HEAD) | `git show :3:f.txt` | Knowing this turns conflict resolution from **guessing into comparing**. ```bash # What did they actually change? Diff their side against the ancestor. $ git diff $(git merge-base HEAD MERGE_HEAD) MERGE_HEAD -- f.txt # Show all three generations (--conflict=diff3 includes the ancestor) $ git checkout --conflict=diff3 -- f.txt ``` It also changes what `git add` means. **`git add` discards stages 1/2/3 and writes a single stage 0 — it is a declaration that you resolved the conflict.** That is exactly why typing `git add .` without reading the result is dangerous. --- ## 5. Deriving `reset` and `restore` from the three trees Here are the three trees we have met. | Tree | Where it lives | What it represents | |---|---|---| | HEAD | `.git/HEAD` → branch → commit | "The last state you committed" | | Index | `.git/index` | "The state you intend to commit next" | | Working tree | The filesystem | "The state you are editing right now" | `git reset` is the command for choosing *how many of these trees* to roll back. | Command | HEAD | Index | Working tree | Typical use | |---|---|---|---|---| | `git reset --soft ` | ✅ moves | ❌ | ❌ | Redo the last commit (changes stay staged) | | `git reset --mixed ` (default) | ✅ moves | ✅ moves | ❌ | Unstage | | `git reset --hard ` | ✅ moves | ✅ moves | ✅ **moves** | Roll back completely (**uncommitted changes are lost**) | | `git restore --staged ` | ❌ | ✅ moves | ❌ | Unstage one file | | `git restore ` | ❌ | ❌ | ✅ moves | Discard edits to one file | | `git switch ` | ✅ moves | ✅ moves | ✅ moves | Switch branches (aborts if changes would conflict) | Git v2.23 split `switch` and `restore` out of `checkout` precisely because those rows were all crammed into one command. **In new documentation and scripts, use `switch` and `restore`.** "Only `--hard` destroys the working tree" — that single line prevents most accidents. Put the other way round: `--soft` and `--mixed` **lose nothing**. The objects remain and so does the reflog. --- ## 6. Inside a merge: merge-base and the ort strategy ### A three-way merge is a comparison against the ancestor Comparing two branches alone cannot tell you whether a line was added on one side or deleted on the other. So Git uses the **common ancestor (merge base)** as a third reference point. ```bash $ git merge-base main feature c3dcd69263c50d4a3fa0fac8428f07fdf1dd27ba ``` For each line: - Ancestor matches `ours`, `theirs` differs → take `theirs` - Ancestor matches `theirs`, `ours` differs → take `ours` - Both differ → **conflict** (three generations land in stages 1/2/3) ### The default strategy, `ort` Quoting `merge-strategies(7)` directly: > `ort` — This is the default merge strategy when pulling or merging one branch. This strategy can only resolve two heads using a 3-way merge algorithm. **When there is more than one common ancestor that can be used for 3-way merge, it creates a merged tree of the common ancestors and uses that as the reference tree for the 3-way merge.** […] The name for this algorithm is an acronym ("Ostensibly Recursive's Twin") and came from the fact that it was written as a replacement for the previous default algorithm, `recursive`. "More than one common ancestor" is the **criss-cross merge** case. When branches merge into each other, the common ancestor is no longer unique, and ort builds a **virtual base tree** by recursively merging the ancestors. Most of the cases where your intuition about "why did this conflict / why didn't it" fails live right here. On `recursive`, the same document is explicit: > This is now a synonym for `ort`. It was an alternative implementation until v2.49.0, but was redirected to mean `ort` in v2.50.0. **Older articles telling you to pass `-s recursive` are advising a no-op.** ### Behaviors and traps worth knowing The documentation warns about one of its own behaviors: > With the strategies that use 3-way merge (including the default, `ort`), if a change is made on both branches, but later reverted on one of the branches, that change will be present in the merged result; some people find this behavior confusing. It occurs because only the heads and the merge base are considered when performing a merge, not the individual commits. "A change I reverted came back through the merge" is **specified behavior, not a bug**. If you revert on a shared branch, the branch merging it may need its own revert. Strategy options worth remembering: | Option | When to reach for it | |---|---| | `-X ours` / `-X theirs` | Auto-resolve conflicting hunks in favor of one side (**not the same as the `-s ours` strategy**, which never looks at the other side at all) | | `-X renormalize` | Kills mass conflicts when the two branches disagree on line-ending normalization or clean filters | | `-X diff-algorithm=patience` | Avoids mismerges caused by meaningless matching lines such as closing braces (**ort defaults to `histogram`**) | | `-X find-renames=` | Tune the rename-detection threshold (rename detection is on by default) | And **if you are resolving the same conflict over and over**, enable `rerere`. It records your resolution and reapplies it when the same conflict recurs — a large win when you repeatedly rebase a long-lived branch. ```bash $ git config --global rerere.enabled true ``` --- ## 7. What rebase and force push really do, and how to use `--force-with-lease` ### Rebase creates new commits As section 2 showed, a commit ID is the hash of content that includes the parent. So **rebase does not move commits; it creates new ones carrying the same changes**. The originals become unreachable and live on in the reflog. That is the real reason behind "don't rebase a shared branch." Other people hold the old IDs you just abandoned, and your push is incompatible with them. ### How `--force-with-lease` works, and the limit the docs admit `git-push(1)` rewards careful reading. > This option overrides this restriction if the current value of the remote ref is the expected value. `git push` fails otherwise. […] It is like taking a "lease" on the ref without explicitly locking it, and the remote ref is updated only if the "lease" is still valid. The question is where the expected value comes from. > `--force-with-lease` alone, without specifying the details, will protect all remote refs that are going to be updated by requiring their current value **to be the same as the remote-tracking branch we have for them**. So the expected value is `refs/remotes/origin/*`. And the documentation states the following as **a general note on safety**: > supplying this option without an expected value, i.e. as `--force-with-lease` or `--force-with-lease=` interacts very badly with anything that implicitly runs `git fetch` on the remote to be pushed to in the background, e.g. `git fetch origin` on your repository in a cronjob. […] The protection it offers over `--force` is ensuring that subsequent changes your work wasn't based on aren't clobbered, but this is **trivially defeated** if some background process is updating refs in the background. Editor extensions and IDEs that fetch periodically are common. So **automation (CI, scripts) should use the explicit form**. ```bash # Good: you declare the remote value you expect $ EXPECTED=$(git rev-parse refs/remotes/origin/main) $ git push --force-with-lease="main:${EXPECTED}" origin main # Stronger: also require that the remote tip is in your own reflog $ git push --force-with-lease --force-if-includes origin main ``` `--force-if-includes` is a complementary check. Per the docs: > This option enables a check that verifies if the tip of the remote-tracking ref is reachable from one of the "reflog" entries of the local branch based in it for a rewrite. The check ensures that any updates from the remote have been incorporated locally by rejecting the forced update if that is not the case. There is a caveat, also stated: **passing it without `--force-with-lease`, or alongside `--force-with-lease=:`, is a no-op.** The first has nothing to build on; the second already declares the expectation explicitly, so no further inference is needed. ### Where each option belongs in practice | Situation | Recommendation | |---|---| | Your own topic branch | `git push --force-with-lease` | | Forced updates from CI/automation | `--force-with-lease=:` with an explicit expectation | | Shared or protected branches | Forbid force updates **on the server** (branch protection rules) | | A shared branch whose history must be fixed | Open a new branch and a PR; do not force push | Rather than banning `--force` outright, **block it server-side and make `--force-with-lease` the client default**. This is a place to rely on mechanism, not human discipline. --- ## 8. Packfiles: how transfer and storage actually work ### From loose objects to packs Everything so far — one object, one file — is the **loose object** format. It becomes inefficient in bulk, so Git periodically consolidates into **packfiles**. ```bash # 13 loose objects $ git count-objects -vH count: 13 size: 52.00 KiB in-pack: 0 packs: 0 size-pack: 0 bytes $ git gc -q # One pack, and much smaller $ git count-objects -vH count: 0 size: 0 bytes in-pack: 13 packs: 1 size-pack: 2.16 KiB ``` 52.00 KiB → 2.16 KiB. This is a three-commit toy repository, so do not generalize that ratio — but the underlying property holds everywhere: **loose objects are rounded up to the filesystem's block size** and therefore occupy far more space than their real size. ### The format (`gitformat-pack(5)`) A `.pack` file is structured as: - A 4-byte signature `{'P','A','C','K'}` - A 4-byte version number (**Git accepts version 2 or 3 but generates version 2 only**) - A 4-byte object count - A run of object entries - A trailing checksum over the whole pack There are seven entry types (one of which is reserved). | Value | Type | |---|---| | 1 | `OBJ_COMMIT` | | 2 | `OBJ_TREE` | | 3 | `OBJ_BLOB` | | 4 | `OBJ_TAG` | | 6 | `OBJ_OFS_DELTA` (base identified by **relative offset** within the same pack) | | 7 | `OBJ_REF_DELTA` (base identified by **object name**) | Types 6 and 7 are **delta compression** — the physical answer to "how can snapshots not explode in size." Similar objects are stored as a base plus delta instructions, and zlib is applied on top. The actual delta chains are visible with `verify-pack`. ```bash $ git verify-pack -v .git/objects/pack/pack-*.idx 719d02e... commit 265 184 12 766195b... commit 67 78 196 1 719d02e... faf9142... commit 19 30 274 2 766195b... ``` The rightmost number is the **delta depth**, followed by the base object. `766195b` is a depth-1 delta against `719d02e`; `faf9142` is depth 2 on top of that. A 67-byte commit is stored in 30 bytes. ### The `.idx` file and the multi-pack index Scanning a pack linearly would defeat the purpose, so each pack ships an `.idx`. The v2 layout is: - The magic number `\377tOc` and version 2 - A **256-entry fan-out table** (cumulative counts by the first byte of the object name) - A sorted object-name table - A CRC32 per object (new in v2 — corruption detection for transfer and storage) - A 4-byte offset table (offsets past 2 GiB spill into an 8-byte large-offset table) - The pack checksum and the index's own checksum The fan-out table narrows the search space to 1/256 using a single byte; a binary search follows. **Even at millions of objects, that is a few dozen comparisons.** When there are many packs, searching across `.idx` files becomes the bottleneck, which is what the **multi-pack-index (MIDX)** solves. Its signature is `{'M','I','D','X'}`, and it is organized into chunks such as `PNAM` (pack names), `OIDF` (fan-out), `OIDL` (name lookup), and `OOFF` (offsets). ### commit-graph: an index for history traversal Reading parent-child relationships out of commit objects on every traversal makes `git log --graph` and reachability checks slow. The **commit-graph** file indexes them separately. Add `--changed-paths` and Git also writes **Bloom filters of changed paths**. > With the `--changed-paths` option, compute and write information about the paths changed between a commit and its first parent. This operation can take a while on large repositories. It provides significant performance gains for getting history of a directory or a file with `git log -- `. In a monorepo where `git log -- path/to/file` crawls, this is usually the single highest-impact change available. --- ## 9. Practical answers for huge repositories ### Partial clone: fetch objects lazily The official design note states the motivation plainly: > During clone and fetch operations, Git downloads the complete contents and history of the repository. […] For extremely large repositories, clones can take hours (or days) and consume 100+GiB of disk space. A partial clone defers fetching objects until they are needed. A remote that can supply them later is called a **promisor remote**. ```bash # Clone without historical blobs (fetched only when a file is actually opened) $ git clone --filter=blob:none https://github.com/org/repo.git # Without trees either (for CI builds and similar, where history traversal is not needed) $ git clone --filter=tree:0 --depth=1 https://github.com/org/repo.git ``` There is an **important precondition**, also stated officially: > Use of partial clone requires that the user be online and the origin remote or other promisor remotes be available for on-demand fetching of missing objects. Offline work or a flaky network turns an innocuous `git log -p` into a storm of extra fetches, making things slower. **This is a choice per use case, not something to impose on everyone.** ### sparse-checkout (cone mode) sparse-checkout narrows the working-tree side. Cone mode trades path-level flexibility for directory-level restriction and speed, and combines with the sparse index described earlier. ```bash $ git sparse-checkout set --cone apps/web packages/ui $ git config core.sparseCheckoutCone true ``` ### `git maintenance`: make upkeep a mechanism Nobody keeps running `git gc` by hand. `git maintenance` is a per-task scheduler. ```bash $ git maintenance start # register with the OS scheduler and run periodically ``` The default schedule for the `incremental` strategy, per `git-maintenance(1)`: | Task | What it does | Default schedule | |---|---|---| | `gc` | Repack everything into a single pack | **Disabled** (in the incremental strategy) | | `commit-graph` | Incrementally update and verify the commit-graph | Hourly | | `prefetch` | Pre-fetch the latest objects from all remotes into `refs/prefetch/` | Hourly | | `loose-objects` | Move loose objects into packs in batches | Daily | | `incremental-repack` | Repack using the multi-pack-index | Daily | | `pack-refs` | Collect loose refs into a single file | Not scheduled (must be enabled explicitly) | | `reflog-expire` | Delete expired reflog entries | Not scheduled | | `rerere-gc` | Clean up the rerere cache | Not scheduled | | `worktree-prune` | Delete stale or broken worktrees | Not scheduled | The `incremental` strategy actually schedules only **those first five** (`gc` is explicitly listed as "disabled"). `pack-refs` and below exist as tasks but are not part of the default schedule. Enable them explicitly if you need them. ```bash # Put pack-refs on a weekly schedule in a ref-heavy repository $ git config maintenance.pack-refs.enabled true $ git config maintenance.pack-refs.schedule weekly ``` `gc` being disabled in the `incremental` strategy is deliberate. **A full repack is expensive and it deletes unreachable objects**, so day-to-day upkeep is better served by the incremental `loose-objects` and `incremental-repack` tasks. ### Practical CI patterns Git in CI has different requirements from Git on a developer's machine: **usually you need the working tree and not the history.** ```yaml # GitHub Actions: jobs that don't need history at all (build, test) - uses: actions/checkout@v5 with: fetch-depth: 1 # Jobs that only need to know which files changed (diff-based lint, impact analysis) # → history metadata is required, past file contents are not - uses: actions/checkout@v5 with: fetch-depth: 0 filter: blob:none ``` `fetch-depth: 0` (full history) is **the most expensive setting by default**. Adding `filter: blob:none` keeps all commits and trees while skipping historical file contents, cutting transfer by an order of magnitude on large repositories. --- ## 10. Integrity and the future: SHA-1, SHA-256, and Git 3.0 ### Today's SHA-1 is not plain SHA-1 `hash-function-transition` records: > Git v2.13.0 and later subsequently moved to a hardened SHA-1 implementation by default, which isn't vulnerable to the SHAttered attack. That is **sha1dc** (SHA-1 with collision detection): it recognizes the bit patterns characteristic of collision attacks and rejects them. So "the SHAttered PDFs can be smuggled into Git" does not apply to current Git. The same document immediately adds that SHA-1 is still weak. ### SHA-256 works, but is not yet a production choice ```bash $ git init --object-format=sha256 myrepo ``` `git-init(1)` is candid: > The valid values are `sha1` and (if enabled) `sha256`. `sha1` is the default. Note: **At present, there is no interoperability between SHA-256 repositories and SHA-1 repositories.** And `hash-function-transition` adds an operational warning: > Until Git protocol gains SHA-256 support, using SHA-256 based storage on public-facing Git servers is strongly discouraged. **Conclusion: starting a new project on SHA-256 today is premature.** Waiting for hosting providers and the toolchain (JGit, libgit2, gitoxide) to catch up is the correct call. ### What changes in Git 3.0 The official `BreakingChanges.adoc` lists five headline changes (with no release date announced). | Change | Detail | Practical impact | |---|---|---| | Default hash | `sha1` → **`sha256`** | New repositories only; the docs state there is no plan to deprecate `sha1` | | Default ref storage | `files` → **`reftable`** | Scripts and hooks that read `.git/refs/*` directly will break | | Default branch name | `master` → **`main`** | Assumptions baked into CI config and docs | | Build requirement | **Rust becomes mandatory** (2.55 default-enables it in both build systems; 3.0 requires it) | Affects self-built and embedded distributions | | `safe.bareRepository` | `all` → **`explicit`** | A security hardening: implicit discovery of bare repositories is refused | The last one deserves the official explanation in full: > It is all too easy for an attacker to trick a user into cloning a repository that contains an embedded bare repository with malicious hooks configured. If the user enters that subdirectory and runs any Git command, Git discovers the bare repository and the hooks fire. **The user does not even need to run a Git command explicitly**: many shell prompts run `git status` in the background to display branch and dirty state information, and `git status` in turn may invoke the fsmonitor hook if so configured, making the user vulnerable the moment they `cd` into the directory. **Arbitrary code execution from a `cd`** is something every day-to-day Git user should know about now. You do not have to wait for Git 3.0. ```bash $ git config --global safe.bareRepository explicit ``` Bare repositories specified explicitly via `--git-dir` or `GIT_DIR` keep working, so the operational side effects are close to zero. **If you ever clone repositories you don't fully trust (which is nearly everyone), this is safe to turn on today.** --- ## 11. Production checklist Turning the internals into settings and practice. ### Safe for everyone ```bash # Prevent implicit hook execution via a malicious bare repository (Git 3.0's default, early) git config --global safe.bareRepository explicit # Record and replay conflict resolutions git config --global rerere.enabled true # A saner push default (understanding the limits from §7) git config --global push.default simple # Detect corrupt objects on the way in and out git config --global transfer.fsckObjects true git config --global fetch.fsckObjects true git config --global receive.fsckObjects true ``` ### Effective on large repositories ```bash # Cache untracked-file discovery (the UNTR extension) git config core.untrackedCache true # Cut stat traversal using filesystem change notifications (the FSMN extension) git config core.fsmonitor true # Register periodic maintenance with the OS scheduler git maintenance start ``` ### What to look for in review | Concern | How to check | Why | |---|---|---| | Large binaries in history | Inspect the largest entries via `git count-objects -vH` / `git verify-pack -v` | Once in, they ride along in every clone forever (deltas don't help) | | Secrets committed | Scan the entire history, not the tip | **The objects remain**, so deleting them in a later commit achieves nothing | | Can anyone force push a shared branch? | Server-side branch protection | Client settings depend on discipline | | Is CI checking out more than it needs? | The `fetch-depth` and `filter` settings | A full-history clone costs jobs × repository size in transfer | | Are hooks trustworthy? | `core.hooksPath` and `safe.bareRepository` | Hooks are arbitrary code execution | --- ## 12. Conclusion: from memorizing to deriving Boiled down, Git's internals are three lines. 1. **The object database is an append-only key-value store keyed by content hash.** Nothing you write disappears (until GC). That is why recovery is possible. 2. **A ref is a 41-byte pointer into that store.** Cheap branching, rebase-as-recreation, and the danger of force push all follow from this one fact. 3. **The index is a draft of the next commit.** It doubles as a `stat` cache, which is why `status` is fast, and it holds three generations at once (stages 1/2/3) during a conflict. With those three in hand, you know which tree you are about to destroy *before* you press enter. Git accidents are, almost without exception, **accidents in which only the working tree is lost** — the objects are still there. And the set of commands that destroy the working tree is small. The natural next step is **the recovery procedures for when something is already lost**: how long the reflog lives, the grace period for unreachable objects (two weeks by default), cruft packs, and `git fsck --lost-found`. Knowing what survives and for how long is what keeps you calm when it happens.