Git 1.7.12

I just tagged the 1.7.12 final release, which is available in tarball form here, and from the usual copies of my repository.

Some highlights:

• An experimental support for UTF-8 pathnames on MacOS.
  I think there still are places that need conversion between the system encoding (UTF-8 normalized by decomposing) and the more commonly used encoding (precomposed) that is used internally for better interoperability, but this should be a good start.
• $HOME/.gitconfig file can be moved to $HOME/.config/git/config, in line with XDG.
  This will also allow $HOME/.config/git/attributes and $HOME/.config/git/ignore, if exist, to be automatically used as core.attributesfile and core.excludesfile, respectively.
• "git apply" learned the same three-way merge patch wiggling magic "git am" supports, via the "-3" option.
• "git rebase -i --root" learned how to update the root commit when requested.
• "git status" can give a more detailed explanation during "intermediate" state of multi-step operations, e.g. "merge", "rebase".
• A remote "SCM" interface to MediaWiki in contrib/ learned to handle file attachments.
• "git clone --no-local $path" bypasses the "local directory?  Just cp -R it!" (not quite that aggressive, but still) optimization and invokes the usual object transfer and repacking codepath.
• The HTMLified documentation pages shown by "git help -w $cmd" could be obtained over the network by setting the help.htmlpath configuration variable.

Enjoy.

Git 1.7.11.5 and Git 1.7.12-rc3

The final release candidate for 1.7.12 is found at the usual places. Again, nothing earth-shattering.

For highlights of the upcoming release, please see the previous entry on 1.7.12-rc1.

Also there is an updated maintenance release 1.7.11.5 that fixes some minor build issues, with updates to "git-blame.el" (in contrib/).

Leftover bits

While developing for Git, sometimes I find a topic getting "stuck" for various reasons, e.g. the basic direction is good, but the remainder of the system is not ready for such a change.

Instead of running "git branch -D topic" in such a case, I use a "git hold" alias that looks like this:

[alias]
  hold = "!sh -c 'git update-ref refs/hold/$1 refs/heads/$1 && \
          git branch -D $1' -"

to stash it away. And "git for-each-ref refs/hold/" gives me many such "held" topics.

Of course, without an active effort to go back, reconsider and prune from time to time, these things tend to accumulate. And today was a good day to review them for me. I found a couple of topics that may still be viable as my reward:

• [DONE - will be in Git 2.0] Make "git add " notice removals of files in the named directories; after all "add" is not about "add files to git", but is "add the current state to the index", and the lack of files that previously existed is part of that state.
  Cf. $gmane/171811
  
  
• [DONE] Allow "git log .." to mean "Give me a log of recent commits that touch my parent directory and what are under it". The current code instead insists that ".." could be both a revision range and a path and asks the user to disambiguate.
  Cf. $gmane/172619

I did not find major downsides on either of these topics after re-thinking the issues, and I didn't see any valid objections from people whose judgement I trust in the old discussions, either.

Perhaps I should try resurrecting them and see what happens.

Git 1.7.12-rc2

An updated release candidate is found at the usual places. You should find nothing earth-shattering in there; just a bunch of translation updates, build clean-ups, and minor documentation updates.

For highlights of the upcoming release, please see the previous entry on 1.7.12-rc1.

Bringing a bit more sanity to "alternates"?

The "alternates" mechanism lets you keep a single object store (not necessarily a git repository on its own, but just the objects/ part of it) on a machine, have multiple repositories on the same machine share objects from it, to save the network transfer bandwidth when cloning from remote repositories and the disk space used by the local repositories.  A repository created by "clone --reference" or "clone -s" uses this mechanism to borrow objects from the object store of another repository.  A user also can manually add new entries to $GIT_DIR/objects/info/alternates to borrow from other object stores.

The UI for this mechanism however has some room for improvement, and we may want to start improving it for the next release after the upcoming Git 1.7.12 (or even Git 2.0 if the change is a large one that may be backward incompatible but gives us a vast improvement).

Here are some random thoughts as a discussion starter (the real discussion is on the git mailing list git@vger.kernel.org; see http://thread.gmane.org/gmane.comp.version-control.git/202902).

By design, the borrowed object store MUST not ever lose any object from it, as such an object loss can corrupt the borrowing repositories.  In theory, it is OK for the object store whose objects are borrowed by repositories to acquire new objects, but losing existing objects is an absolute no-no.

But the UI of "clone -s" encourages users to borrow from the object store of a repository that the user may actively develop in.  It is perfectly normal for users to perform operations that make objects that used to be reachable from tips of its branches unreachable (e.g. rebase, reset, "branch -d") in a repository that is used for active development, but a "gc" after such an operation will lose objects that were originally available in the repository.  If objects lost that way were still needed by the repositories that borrow from it, the borrowing repository gets corrupt immediately.

In practice, this means that users who use "clone -s" to make a new repository can *never* prune the original repository without risking to corrupt its borrowing repository [1].
Some ideas:

• Make "clone --reference" without "-s" not to borrow from the reference repository.  E.g. if you have a clone of Linus repository at /git/linux.git/, cloning a related repository using it as --reference:
  
  $ git clone --reference /git/linux.git git://k.org/linux-next.git should still take advantage of /git/linux.git/{refs,objects} to reduce the transfer cost of fetching from k.org, but the resulting repository should not point /git/linux.git with its objects/info/alternates file.
• Make the distinction between a regular repository and an object store that is meant to be used for object sharing stronger.
  Perhaps a configuration item "core.objectstore = readonly" can be introduced, and we forbid "clone -s" from pointing at a repository without such a configuration.  We also forbid object pruning operations such as "gc" and "repack" from being run in a repository marked as such.
  
  It may be necessary to allow some special kind of repacking of such a "readonly" object store, in order to reduce the number of packfiles (and get rid of loose object files); it needs to be implemented carefully not to lose any object, regardless of local reachability.

When you have a repository and one or more repositories that borrow from it, you may want to dissociate the borrowing repositories from the borrowed one (e.g. so that you can repack or prune the original repository safely, or you may even want to remove it).

I think "git repack -a -d [-f]" in the borrowing repository happens to be the way to do this, but it is not clear to the users why.

Some ideas:

• It might not be a bad idea to have a dedicated new command to help users manage alternates ("git alternates"?); obviously this will be one of its subcommand "git alternates detach" if we go that route.
• Or just an entry in the documentation is sufficient?

When you have two or more repositories that do not share objects, you may want to rearrange things so that they share their objects from a single common object store.

There is no direct UI to do this, as far as I know.  You can obviously create a new bare repository, push there from all of these repositories, and then borrow from there, e.g.

git --bare init shared.git &&
for r in a.git b.git c.git ...
        do
   (
cd "$r" &&
       git push ../shared.git "refs/*:refs/remotes/$r/*" &&
echo ../../../shared.git/objects >.git/objects/info/alternates
       )
done

And then repack shared.git once.

Some ideas:

• (obvious: give a canned command to do the above, perhaps then set the core.objectstore=readonly in the resuting shared.git)

When you have one object store and a repository that does not yet borrow from it, you may want to make the repository borrow from the object store.  Obviously you can run "echo" like the sample script in the previous item above, but it is not obvious how to perform the logical next step of shrinking $GIT_DIR/objects of the repository that now borrows the objects.

[edit: This is supported as "git repack -a -d -l"]

I think "git repack -a -d" is the way to do this, but if you compare this command to "git repack -a -d -f" we saw previously in this message, it is not surprising that the users would be confused---it is not obvious at all.

Some ideas:

• (obvious: give a canned subcommand to do this)

[Footnote]

1 Making the borrowed object store aware of all the repositories that borrow from it, so that operations like "gc" and "repack" in the repository with the borrowed object store can keep objects that are needed by borrowing repositories, is theoretically possible, but is not a workable approach in practice, as (1) borrowers may not have a write access to the shared object store to add such a back pointer to begin with, (2) "gc"/"repack" in the borrowed object store and normal operations in the borrowing repositories can easily race with each other, without any coordination between the users, and (3) a casual "borrowing" can simply be done with a simple "echo" as shown in the main text of this message, and there is no way to ensure a backpointer from the borrowed object store to such a borrowing repository.

Git 1.7.12-rc1

The first release candidate for the upcoming Git 1.7.12 can be found at

    http://code.google.com/p/git-core/downloads/list

and also at the usual repositories.

Some highlights:

• An experimental support for UTF-8 pathnames on MacOS.
  I think there still are places that need conversion between the system encoding (UTF-8 normalized by decomposing) and the more commonly used encoding (precomposed) that is used internally for better interoperability, but this should be a good start.
• $HOME/.gitconfig file can be moved to $HOME/.config/git/config, in line with XDG.
  This will also allow $HOME/.config/git/attributes and $HOME/.config/git/ignore, if exist, to be automatically used as core.attributesfile and core.excludesfile, respectively.
• "git apply" learned the same three-way merge patch wiggling magic "git am" supports, via the "-3" option.
• "git rebase -i --root" learned how to update the root commit when requested.
• "git status" can give a more detailed explanation during "intermediate" state of multi-step operations, e.g. "merge", "rebase".
• A remote "SCM" interface to MediaWiki in contrib/ learned to handle file attachments.

Please test so that we can avoid regressions in the final release that will happen late August.

Thanks.

Git 1.7.11.4

The fourth maintenance release for 1.7.11.x series has been tagged and is available at

    http://code.google.com/p/git-core/downloads/list

It updates 1.7.11.3 with a handful of updates. The most notable one may be that you can now run filter-branch across a commit whose committer timestamp is in year 1970 (more precisely, the UNIX epoch).

Git 1.7.12-rc0

A release-candidate preview for the upcoming 1.7.12 has been tagged. The release tarballs are found at:

    http://code.google.com/p/git-core/downloads/list

as usual.

Git v1.7.12 Release Notes (draft)

Updates since v1.7.11

UI, Workflows & Features

• Git can be told to normalize pathnames it read from readdir(3) and all arguments it got from the command line into precomposed UTF-8 (assuming that they come as decomposed UTF-8), in order to work around issues on Mac OS.
• I think there still are other places that need conversion (e.g. paths that are read from stdin for some commands), but this should be a good first step in the right direction.
• Per-user $HOME/.gitconfig file can optionally be stored in $HOME/.config/git/config instead, which is in line with XDG.
• The value of core.attributesfile and core.excludesfile default to $HOME/.config/attributes and $HOME/.config/ignore respectively when these files exist.
• Logic to disambiguate abbreviated object names have been taught to take advantage of object types that are expected in the context, e.g. XXXXXX in the "git describe" output v1.2.3-gXXXXXX must be a commit object, not a blob nor a tree.  This will help us prolong the lifetime of abbreviated object names.
• "git apply" learned to wiggle the base version and perform three-way merge when a patch does not exactly apply to the version you have.
• Scripted Porcelain writers now have access to the credential API via the "git credential" plumbing command.
• "git clone --local $path" started its life as an experiment to optionally use link/copy when cloning a repository on the disk, but we didn't deprecate it after we made the option a no-op to always use the optimization.  The command learned "--no-local" option to turn this off, as a more explicit alternative over use of file:// URL.
• "git fetch" and friends used to say "remote side hung up unexpectedly" when they failed to get response they expect from the other side, but one common reason why they don't get expected response is that the remote repository does not exist or cannot be read. The error message in this case was updated to give better hints to the user.
• "git help -w $cmd" can show HTML version of documentation for "git-$cmd" by setting help.htmlpath to somewhere other than the default location where the build procedure installs them locally; the variable can even point at a http:// URL.
• "git rebase [-i] --root $tip" can now be used to rewrite all the history leading to "$tip" down to the root commit.
• "git rebase -i" learned "-x " to insert "exec " after each commit in the resulting history.
• "git status" gives finer classification to various states of paths in conflicted state and offer advice messages in its output.
• "git submodule" learned to deal with nested submodule structure where a module is contained within a module whose origin is specified as a relative URL to its superproject's origin.
• A rather heavy-ish "git completion" script has been split to create a separate "git prompting" script, to help lazy-autoloading of the completion part while making prompting part always available.
• "gitweb" pays attention to various forms of credits that are similar to "Signed-off-by:" lines in the commit objects and highlights them accordingly.

Foreign Interface

• "mediawiki" remote helper (in contrib/) learned to handle file attachments.
• "git p4" now uses "Jobs:" and "p4 move" when appropriate.
• vcs-svn has been updated to clean-up compilation, lift 32-bit limitations, etc.

Performance, Internal Implementation, etc. (please report possible regressions)

• Some tests showed false failures caused by a bug in ecryptofs.
• We no longer use AsciiDoc7 syntax in our documentation and favor a more modern style.
• "git am --rebasing" codepath was taught to grab authorship, log message and the patch text directly out of existing commits.  This will help rebasing commits that have confusing "diff" output in their log messages.
• "git index-pack" and "git pack-objects" use streaming API to read from the object store to avoid having to hold a large blob object in-core while they are doing their thing.
• Code to match paths with exclude patterns learned to avoid calling fnmatch() by comparing fixed leading substring literally when possible.
• "git log -n 1 -- rarely-touched-path" was spending unnecessary cycles after showing the first change to find the next one, only to discard it.

Also contains minor documentation updates and code clean-ups.

Fixes since v1.7.11

Unless otherwise noted, all the fixes since v1.7.11 in the maintenance releases are contained in this release (see release notes to them for details).

• "git commit-tree" learned a more natural "-p " order of arguments long time ago, but recently forgot it by mistake.  (merge 4b7518a kk/maint-commit-tree later to maint).
• "git diff --no-ext-diff" did not output anything for a typechange filepair when GIT_EXTERNAL_DIFF is in effect.  (merge c12f82a jv/maint-no-ext-diff later to maint).
• When "git am" failed, old timers knew to check .git/rebase-apply/patch to see what went wrong, but we never told the users about it.  (merge 14bf2d5 pg/maint-1.7.9-am-where-is-patch later to maint).
• When "git submodule add" clones a submodule repository, it can get confused where to store the resulting submodule repository in the superproject's .git/ directory when there is a symbolic link in the path to the current directory.  (merge 6eafa6d jl/maint-1.7.10-recurse-submodules-with-symlink later to maint).
• In 1.7.9 era, we taught "git rebase" about the raw timestamp format but we did not teach the same trick to "filter-branch", which rolled a similar logic on its own.  (merge 44b85e89 jc/maint-filter-branch-epoch-date later to maint).

Git 1.7.11.3

The third maintenance release for 1.7.11 has been tagged and is available at the usual places (see Git Public Repositories link).

• The error message from "git push $there :bogo" (and its equivalent "git push $there --delete bogo") mentioned that we tried and failed to guess what ref is being deleted based on the LHS of the refspec, which we don't.
• A handful of files and directories we create had tighter than necessary permission bits when the user wanted to have group writability (e.g. by setting "umask 002").
• "commit --amend" used to refuse amending a commit with an empty log message, with or without "--allow-empty-message".
• "git commit --amend --only --" was meant to allow "Clever" people to rewrite the commit message without making any change even when they have already changes for the next commit added to their index, but it never worked as advertised since it was introduced in 1.3.0 era.
• Even though the index can record pathnames longer than 1<<12 bytes, in some places we were not comparing them in full, potentially replacing index entries instead of adding.
• "git show"'s auto-walking behaviour was an unreliable and unpredictable hack; it now behaves just like "git log" does when it walks.
• "git diff", "git status" and anything that internally uses the comparison machinery was utterly broken when the difference involved a file with "-" as its name.  This was due to the way "git diff --no-index" was incorrectly bolted on to the system, making any comparison that involves a file "-" at the root level incorrectly read from the standard input.
• We did not have test to make sure "git rebase" without extra options filters out an empty commit in the original history.
• "git fast-export" produced an input stream for fast-import without properly quoting pathnames when they contain SPs in them.
• "git checkout --detach", when you are still on an unborn branch, should be forbidden, but it wasn't.
• Some implementations of Perl terminates "lines" with CRLF even when the script is operating on just a sequence of bytes.  Make sure to use "$PERL_PATH", the version of Perl the user told Git to use, in our tests to avoid unnecessary breakages in tests.

The release tarballs are found at:

http://code.google.com/p/git-core/downloads/list

On the 'master' front, we are getting very close to the next feature release 1.7.12.

Git 1.7.11.2

The second maintenance release for 1.7.11 has many real fixes (the first one was mainly to fix the HTML formatted documentation pages) to bugs identified, diagnosed and fixed in the past three weeks. These fixes have been cooking on the master front for some time.

• On Cygwin, the platform pread(2) is not thread safe, just like our own compat/ emulation, and cannot be used in the index-pack program.  Makefile variable NO_THREAD_SAFE_PREAD can be defined to avoid use of this function in a threaded program.
• "git add" allows adding a regular file to the path where a submodule used to exist, but "git update-index" does not allow an equivalent operation to Porcelain writers.
• "git archive" incorrectly computed the header checksum; the symptom was observed only when using pathnames with hi-bit set.
• "git blame" did not try to make sure that the abbreviated commit object names in its output are unique.
• Running "git bundle verify" on a bundle that records a complete history said "it requires these 0 commits".
• "git clone --single-branch" to clone a single branch did not limit the cloning to the specified branch.
• "git diff --no-index" did not correctly handle relative paths and did not correctly give exit codes when run under "--quiet" option.
• "git diff --no-index" did not work with pagers correctly.
• "git diff COPYING HEAD:COPYING" gave a nonsense error message that claimed that the treeish HEAD did not have COPYING in it.
• When "git log" gets "--simplify-merges/by-decoration" together with "--first-parent", the combination of these options makes the simplification logic to use in-core commit objects that haven't been examined for relevance, either producing incorrect result or taking too long to produce any output.  Teach the simplification logic to ignore commits that the first-parent traversal logic ignored when both are in effect to work around the issue.
• "git ls-files --exclude=t -i" did not consider anything under t/ as excluded, as it did not pay attention to exclusion of leading paths while walking the index.  Other two users of excluded() are also updated.
• "git request-pull $url dev" when the tip of "dev" branch was tagged with "ext4-for-linus" used the contents from the tag in the output but still asked the "dev" branch to be pulled, not the tag.

As usual, the release tarballs are found at:

http://code.google.com/p/git-core/downloads/list

and the following public repositories all have a copy of the v1.7.11.2 tag and the maint branch that the tag points at:

url = git://repo.or.cz/alt-git.git

url = https://code.google.com/p/git-core/

url = git://git.sourceforge.jp/gitroot/git-core/git.git

url = git://git-core.git.sourceforge.net/gitroot/git-core/git-core

url = https://github.com/gitster/git