Skip to content

source/git: recover from stale submodule cache after removal#6563

Merged
tonistiigi merged 1 commit into
moby:masterfrom
brianr2600:brian/4260-stale-submodule-cache
Mar 24, 2026
Merged

source/git: recover from stale submodule cache after removal#6563
tonistiigi merged 1 commit into
moby:masterfrom
brianr2600:brian/4260-stale-submodule-cache

Conversation

@brianr2600
Copy link
Copy Markdown
Contributor

When a cached git source has initialized submodules for one commit and a later commit removes a submodule, git submodule update can fail against the reused cached repo with 'No url found for submodule path ... in .gitmodules'.

Detect that specific stale-submodule mismatch during checkout, refetch the remote repo into a fresh cache state, and retry the checkout once. Add a regression test that advances a branch from a commit with a submodule to one that removes it and verifies the second snapshot succeeds for both sha1 and sha256 repositories.

Signed-off-by: Brian Ristuccia brian@ristuccia.com

@brianr2600
Copy link
Copy Markdown
Contributor Author

This pull request likely fixes #4260

Comment thread source/git/source.go Outdated

func (gs *gitSourceHandler) updateSubmodules(ctx context.Context, git *gitutil.GitCLI) error {
args := []string{"submodule", "update", "--init", "--recursive", "--depth=1"}
if _, err := git.Run(ctx, args...); err != nil {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't there some command we could just run in here to fix up the submodule state? Triggering fetch again on checkout is ugly and possibly with some security considerations. Or alternatively something we could run during fetch to make sure submodule state is still ok.

What prevents this from going to a loop?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will run only once. But thanks for pushing back on simply discarding the old snapshot as it prompted me to do some additional digging which revealed the likely root cause behind #4260

The git checkout tree-ish -- pathspec buildkit uses to populate the build snapshot defaults to overlay mode, so it won't delete objects which exist in the work tree but don't exist in the tree-ish. For ordinary file and directory objects, it's not a problem because the destination work tree is a freshly made empty directory. In the case of submodules, deleted or renamed gitlink references continue to exist in the index which is used by the subsequent git submodule update ... command and may cause it to fail.

With a a non-overlay checkout in the happy path, this new error handler becomes unnecessary. I plan to push an update to this PR soon.

Comment thread source/git/source.go Outdated
}

func isMissingSubmoduleURLError(err error) bool {
return strings.Contains(err.Error(), "No url found for submodule path") &&
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this same error can't be just created by some repo configuration problem.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - creating an intentionally mangled gitmodules file or bogus gitlink references in a tree can also trigger this error.

The `git checkout tree-ish -- pathspec` command run by buildkit to
populate the work tree defaults to overlay mode, so it won't delete objects
which exist in the work tree but don't exist in the tree-ish. It's not a
problem for ordinary file and directory objects because the work tree
starts out empty, but in the case where submodules are deleted or
renamed it will leave stale gitlink references in the index. The
subsequent `git submodule update ...` command will then fail with an
eror like 'No url found for submodule path ... in .gitmodules' as seen
in moby#4260.

Adding `--no-overlay` ensures that any deleted gitlink references are
removed from the index before the submodule update runs.

Signed-off-by: Brian Ristuccia <brian@ristuccia.com>
@brianr2600 brianr2600 force-pushed the brian/4260-stale-submodule-cache branch from ce473d3 to ea11d20 Compare March 11, 2026 18:26
@brianr2600
Copy link
Copy Markdown
Contributor Author

brianr2600 commented Mar 11, 2026

This updated pull request addresses the likely root cause of #4260 by adding --no-overlay to the pathspec style checkout in the happy path, ensuring the index is fully updated (including deletions) before git submodules update is run. Note that --no-overlay requires Git >= 2.22.0.

If maintaining compatibility with older versions of git is a requirement, the non-pathspec based git checkout -f <ref> could be used instead. It also removes removes deleted items from the index, but has the (likely harmless) side effect of updating .git/HEAD.

Another alternative is git reset --hard <ref>, which fully updates the index to match <ref>, updates .git/HEAD and .git/ORIG_HEAD, and populates the empty work tree as a side effect.

All three variants appear to fix #4260 and introduced no new regression suite failures.

@tonistiigi tonistiigi merged commit 1fba907 into moby:master Mar 24, 2026
189 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants