The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,094 other followers

“error: invalid object 100644” “git svn”

Posted by jpluimers on 2020/07/14

A while back, while using “git svn”, on a Windows system, I got [Archive.is“error: invalid object 100644” “git svn” – Google Search after statements like this:

# git svn rebase
error: refs/remotes/git-svn does not point to a valid object!
error: invalid object 100644 ac7df132f5bd7d639fc525f1f0204a546658d0c5 for 'Source/ToDoList/GX_ToDo.pas'
fatal: git-write-tree: error building trees
write-tree: command returned error: 128

# git svn fetch
error: refs/remotes/git-svn does not point to a valid object!
error: invalid object 100644 ac7df132f5bd7d639fc525f1f0204a546658d0c5 for 'Source/ToDoList/GX_ToDo.pas'
fatal: git-write-tree: error building trees
write-tree: command returned error: 128

In my case, regular git operations (like branching, committing, pushing, etc) worked fine, but git svn would fail.

One problem was that [Archive.is“error: refs/remotes/git-svn does not point to a valid object” – Google Search only returned one un-meaningful result: [WayBack] gist:87613 · GitHub.

Luckily, I had a backup (though it was from a while ago as that VM had not been in use for quite some time) which is the first part in [WayBack] Git FAQ – Git SCM Wiki: How to fix a broken repo?.

Since I was still interested finding out how to resurrect, just in case this happens at a time the backups do not go back far enough, I tried the steps below.

The very first fixing step is to ensure you can quickly restore things, or even better: operate on a copy of the broken pieces. On Windows, robocopy /mir is my friend for this, in Linux rsync -avloz (although on some systems, -z crashes).

TL;DR from the fixing steps

Find out what problems you have, and in which order to fix them. Otherwise you will break more stuff and take longer to fix it.

In this case, two things failed: one on the git side, and one on the git svn side. Since git svn depends on git, the best approach is to fix the git problem first, then the git svn thing.

Fixing this manually try 1

After reading [WayBack] git svn – fatal: git-write-tree: error building trees – Stack Overflow, I tried fixing this with a git gc , but that made things worse:

# git svn fetch
error: refs/remotes/git-svn does not point to a valid object!
fatal: failed to unpack tree object 1771d47096e5db1f8a7565f91cbe803c128050d5
read-tree 1771d47096e5db1f8a7565f91cbe803c128050d5: command returned error: 128

So I tried checking the repository integrity:

# git fsck --full
error: refs/remotes/git-svn does not point to a valid object!
fatal: failed to unpack tree object 1771d47096e5db1f8a7565f91cbe803c128050d5
read-tree 1771d47096e5db1f8a7565f91cbe803c128050d5: command returned error: 128

This basically means the .git\logs\refs\remotes\git-svn file is broken: restore it from a back-up.

A new try:

# git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (7919/7919), done.
error: refs/remotes/git-svn: invalid sha1 pointer 1771d47096e5db1f8a7565f91cbe803c128050d5
error: refs/remotes/git-svn: invalid reflog entry 1771d47096e5db1f8a7565f91cbe803c128050d5

This was the git hash calculated from the most recent SVN commit, so I decided to cut the last line from the .git\logs\refs\remotes\git-svn file, then have .git\refs\remotes\git-svn point to it (I used Notepad++ for this as it does not fiddle with line ending styles).

Now git fsck --full was happy, but then this happened (same for git svn info):

# git svn log
fatal: Invalid revision range 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn
rev-list --pretty=raw --reverse 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn --: command returned error: 128

Dang: there is another place where git svn stores the last git hash!

I tried repairing this with git gc, but in addition to finding another error, it again also deleted .git\logs\refs\remotes\git-svn in addition to this error:

# git gc
Counting objects: 12484, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2849/2849), done.
Writing objects: 100% (12484/12484), done.
Total 12484 (delta 9591), reused 12484 (delta 9591)
Unlink of file '.git/objects/pack/pack-ad6305bd7175b962a35c0c2d9eb1fd14a5163116.idx' failed. Should I try again? (y/n) n
warning: unable to unlink '.git/objects/pack/pack-ad6305bd7175b962a35c0c2d9eb1fd14a5163116.idx': Invalid argument

I found the error git gc was caused by a git diff still being active as it had started Beyond Compare, even though the comparison window in Beyond Compare was already closed. After closing Beyond Compare, git gc succeeded, but git svn info still failed:

# git gc
Counting objects: 12484, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2849/2849), done.
Writing objects: 100% (12484/12484), done.
Total 12484 (delta 9591), reused 12484 (delta 9591)

# git svn info
fatal: Invalid revision range 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn
rev-list --pretty=raw --reverse 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn --: command returned error: 128

I stopped here, as I felt I was working myself into more trouble, because I had the approach the wrong way.

Fixing this manually try 2

In the first try, the git gc goofed things up.

So the second try is without that step. And it is much shorter!

# git svn fetch
error: refs/remotes/git-svn does not point to a valid object!
error: invalid object 100644 ac7df132f5bd7d639fc525f1f0204a546658d0c5 for 'Source/ToDoList/GX_ToDo.pas'
fatal: git-write-tree: error building trees
write-tree: command returned error: 128

# git svn info
error: refs/remotes/git-svn does not point to a valid object!
Path: .
URL: https://svn.code.sf.net/p/gexperts/code/trunk
Repository Root: https://svn.code.sf.net/p/gexperts/code
Repository UUID: 03fc42d2-69ba-4c40-8a67-5bfb646c2f2c
Revision: 2157
Node Kind: directory
Schedule: normal
Last Changed Author: twm
Last Changed Rev: 2157
Last Changed Date: 2018-01-29 14:37:02 +0100 (Mon, 29 Jan 2018)

The error changes after a check:

# git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (7919/7919), done.
error: refs/remotes/git-svn: invalid sha1 pointer 1771d47096e5db1f8a7565f91cbe803c128050d5
error: refs/remotes/git-svn: invalid reflog entry 1771d47096e5db1f8a7565f91cbe803c128050d5
dangling blob 0d8463cae1b8abbfb7f0393ee66d58ffed143da0
...

This was the git hash calculated from the most recent SVN commit, so I decided to cut the last line from the .git\logs\refs\remotes\git-svn file, then have .git\refs\remotes\git-svn point to it.

Now git fsck --full was happy, but then this happened (same for git svn log):

# git svn info
fatal: Invalid revision range 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn
rev-list --pretty=raw --reverse 1771d47096e5db1f8a7565f91cbe803c128050d5..refs/remotes/git-svn --: command returned error: 128

Knowing the previous time git gc does not work, I still tried it again (as back then it could have been caused by the hanging git diff), but this deleted both the `svn

# git gc
Counting objects: 12474, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6395/6395), done.
Writing objects: 100% (12474/12474), done.
Total 12474 (delta 9583), reused 7719 (delta 6035)
Removing duplicate objects: 100% (256/256), done.

Now there is only one git-svn file left:

# @for /r %a in (git-svn*) do @echo %~fa %~za
...\.git\logs\refs\remotes\git-svn 0

Before the git gc there were two files:

# @for /r %a in (git-svn*) do @echo %~fa %~za
...\.git\logs\refs\remotes\git-svn 234184
...\.git\refs\remotes\git-svn 41

So after restoring to the “before git gc state”, lets try the suggestion from

[WayBack] git svn – git-svn fetch/rebase failing after trying to repair a corrupted file – Stack Overflow that I found via [Archive.is] “fatal: Invalid revision range” “svn” – Google Search and [WayBack] git-svn fetch gives Invalid revision range after bfg cleanup · Issue #185 · rtyley/bfg-repo-cleaner · GitHub:

# dir /s /b .rev_map.*
...\.git\svn\refs\remotes\git-svn\.rev_map.03fc42d2-69ba-4c40-8a67-5bfb646c2f2c

You can find the uuid through this command:

# type .git\svn\.metadata
; This file is used internally by git-svn
; You should not have to edit it
[svn-remote "svn"]
reposRoot = https://svn.code.sf.net/p/gexperts/code
uuid = 03fc42d2-69ba-4c40-8a67-5bfb646c2f2c

Now delete this file (assuming there is just one):

# del /s .git\svn\refs\remotes\git-svn\.rev_map.*
Deleted file - ...\.git\svn\refs\remotes\git-svn\.rev_map.03fc42d2-69ba-4c40-8a67-5bfb646c2f2c

The .rev_map.* file is now actually being re-built:

# git svn info
Rebuilding .git/svn/refs/remotes/git-svn/.rev_map.03fc42d2-69ba-4c40-8a67-5bfb646c2f2c ...
r1 = 2b9cc680064f4c3811209cd32adbf0108585ecb7
...
r2163 = 8c4268f0d8ee2d3fe4d5d059b8499128d8ac0662
Done rebuilding .git/svn/refs/remotes/git-svn/.rev_map.03fc42d2-69ba-4c40-8a67-5bfb646c2f2c
Path: .
URL: https://svn.code.sf.net/p/gexperts/code/trunk
Repository Root: https://svn.code.sf.net/p/gexperts/code
Repository UUID: 03fc42d2-69ba-4c40-8a67-5bfb646c2f2c
Revision: 2157
Node Kind: directory
Schedule: normal
Last Changed Author: twm
Last Changed Rev: 2157
Last Changed Date: 2018-01-29 14:37:02 +0100 (Mon, 29 Jan 2018)

Now the check is happy:

# git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (12474/12474), done.

But fetch is not yet:

# git svn fetch
error: invalid object 100644 ac7df132f5bd7d639fc525f1f0204a546658d0c5 for 'Source/ToDoList/GX_ToDo.pas'
fatal: git-write-tree: error building trees
write-tree: command returned error: 128

The odd thing is that the file does exist:

# dir /s /b GX_ToDo.pas
...\Source\ToDoList\GX_ToDo.pas

I was not sure which hashing git was using (it is moving from sha1 to sha256 [WayBack] cryptography – Why doesn’t Git use more modern SHA? – Stack Overflow), and was not sure how the hashing was performed, so I did some digging and found out to NOT use git hash-object as I found via [WayBack] corruption – How to recover Git objects damaged by hard disk failure? – Stack Overflow:

# git help -a | findstr /i "hash"
askpass hash-object remote-http

It will give you the wrong hash:

# git hash-object Source\ToDoList\GX_ToDo.pas
2169df8d94165fec1eb5379dee54cd0910c921fc

[WayBack] Git – finding the SHA1 of an individual file in the index – Stack Overflow describes how to get the right hash (and attributes as a bonus):

git ls-files -s Source\ToDoList\GX_ToDo.pas
100644 1f9ef139c33c88d2ba90b49c4677cf60910bb67b 0 Source/ToDoList/GX_ToDo.pas

Anyway: the above shows that the current version of Source/ToDoList/GX_ToDo.pas does not match the hash, so adding it has no use. So I used a two step process trying to find the hash of the file during all commits, based on [WayBack] git – List all commits for a specific file – Stack Overflow:

  1. Get a list of all commit-hashes

# git log --follow --format=%H -- Source\ToDoList\GX_ToDo.pas
3325ccd5dcf1b1af22faf186173388ac7fb39039
0a75adc2772e3b5661cb61f42876c531cc5055ad
a0b5dc3b8e6656adb5ec4d9fedfc0e7fbed3f5d1
e4d6479a6c5fc96277f89bf3a711cae8a1251c25
987027a6dcdf36c1a24f2c25ddb9cdd40102baf9
a0e743a8f5cda1e940d5af3f3f01b8c236015db6
555805cec0366a6baaf52e5caa7d34cc26872861
ef076e162d0b6a8daa43ca1f4a5e2064b6253b18
2ebbbc6abfaf8b723a7d6ebe198247b933e112dd
bb37b50f67d3945e93aacccc793d59bf466c77fe
7c0d3109e1da7b1525499475cf0a14586be6a406
aa4af6ed28fc16a2e66280adb95d1a154d389098
d8747626b243bde96a4d9a68011a1d471ba02d36
2add303a48e9531e52a98ce88fae5a2b88a01f99
bc2654ba09495e8642b7ad551dc68a8f5d028126
c3a926a188eb48506254f3eea053d0eaec740c2e
41ee56c649d40a1590ff2630087268e088b2cbf5
c98be9848be2b46b79f1fc6daf60554dbb1d1616
4dfdc352f2fbe67295c001ec47b40a06588294f6
45b2abdbc397b41899147c5106875738ce559974
62448b7a4c88693d6c3c32e867cb13f146dc9782
2b9cc680064f4c3811209cd32adbf0108585ecb7

  1. Generate a batch file from the above list that checks out the commit, then gets the hash for that file, with a command  based on [WayBack] Git – finding the SHA1 of an individual file in the index – Stack Overflow that looks like this:

git checkout 3325ccd5dcf1b1af22faf186173388ac7fb39039 && git ls-files -s Source\ToDoList\GX_ToDo.pas

Result: No commit had a file hash matching ac7df132f5bd7d639fc525f1f0204a546658d0c5 for  Source\ToDoList\GX_ToDo.pas.

Having bumped into the git gc issue killing the git-svn files, I got shy of [WayBack] git svn – fatal: git-write-tree: error building trees – Stack Overflow, but re-read it.

That made me understand there are two “garbage collectors:

  1. [WayBack] Git – git-gc Documentation
  2. [WayBack] Git – git-svn Documentation: gc

The first damages the git-svn files from the second. But what if the second could just be ran without influencing the first?

So I gave it a try:

# git svn gc

The command prompt returned without any output, so I was afraid nothing had happened.

But something happened as now I could perform a fetch again, as this worked fine taking about 15 minutes to run:

C:\Users\jeroenp\Versioned\GExperts>git svn fetch
        M       Source/ToDoList/GX_ToDo.pas
r2164 = 1771d47096e5db1f8a7565f91cbe803c128050d5 (refs/remotes/git-svn)
        M       ExternalSource/mPasLex.pas
r2165 = 98f8dd43199ba51895406aca0e2ae29b2a8b36f2 (refs/remotes/git-svn)
        M       ExternalSource/mwPasTokenList.pas
...
r2435 = 9d9b91a82fd283b45e4b111c70ae5389ed086429 (refs/remotes/git-svn)
        M       Source/FavoriteFiles/GX_FavFiles.pas
r2436 = ca41a65cafa612af0208cf98be72c2830d6c44cd (refs/remotes/git-svn)

Part of fixing this manually, try 3

In a third attempt, I tried getting the full repository:

# git svn clone https://svn.code.sf.net/p/gexperts/code/trunk GExperts.git-svn

With that in place, I generated another hash list:

# git log --follow --format=%H -- Source\ToDoList\GX_ToDo.pas > do.bat

Then I edited do.bat so all lines looked like this, but with the correct commit hash at the start:

git checkout 3325ccd5dcf1b1af22faf186173388ac7fb39039 && git ls-files -s Source\ToDoList\GX_ToDo.pas

Then I ran do.bat and verified the output: and indeed there was a commit having Source\ToDoList\GX_ToDo.pas with the failing hash!

This could allow me to copy ...\.git\objects\ac\7df132f5bd7d639fc525f1f0204a546658d0c5 over to the failing repository, as described in [WayBack] git commit stopped working – Error building trees – Stack Overflow.

After some more searching, I also found a much quicker way to get the hashes for Source\ToDoListGX_ToDo.pas:

# git log --raw --abbrev=-1

This gives a commit log where (because of the --raw, see [WayBack] Git – git-log Documentation: –raw) the format is described at [WayBack] Git – git-diff Documentation: diff format for merges:

  1. there is a colon for each parent
  2. there are more “src” modes and “src” sha1
  3. status is concatenated status characters for each parent
  4. no optional “score” number
  5. single path, only for “dst”

Example:

::100644 100644 100644 fabadb8... cc95eb0... 4866510... MM describe.c

By default, the hashes are quite short, but the --abbrev=-1 (though undocumented) extends those to maximum length.

Success

So after the above success, I quickly made a new robocopy backup, so I could finalize the process:

# git svn rebase

# git push origin master:master

Now https://github.com/jpluimers/GExperts.git was up-to-date with https://svn.code.sf.net/p/gexperts/code/trunk again.

–jeroen

Related:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: