Find git commits which still need to be merged

siker — Sat, 29 May 2010 02:14:35 +0000

If you have a git branch which you are working on which is a clone from an upstream ‘official’ branch, and you sometimes submit your commits to be integrated upstream, there’ll surely be times when your patches are not accepted or fall between the cracks. After a while it can get a little hairy to figure out which commits are unique to your branch and which ones are incorporated in the main repo.

The ‘git cherry’ command can help with this. It lists all commits in one branch which are not in another. It works like this simply:

git cherry -v upstream my_branch

The man page describes the command succinctly:

Every commit that doesn’t exist in the branch has its id (sha1) reported, prefixed by a symbol. The ones that have equivalent change already in the branch are prefixed with a minus (-) sign, and those that only exist in the branch are prefixed with a plus (+) symbol.

The problem is that sometimes the cherry command does not identify ‘equivalent changes’. I’m not sure why, but perhaps this happens when the main tree has cherry picked commits and made changes to whitespace or resolved merge conflicts. Or maybe it doesn’t work well at all with cherry pick. Either way, I had a list of changes which cherry returned which I knew had in fact been merged. So I whipped up up this silly little Python script to filter a cherry results list from a complete log of the upstream repo and to remove all commits with the same commit message. This is obviously not bullet proof as different commits may have the same message. But it was good enough for me in this case. So without further ado, here it is:

import sys

cherry = open(sys.argv[1], "r")
filtr = open(sys.argv[2], "r")
lines = filtr.readlines()
unmatched = []

for line in cherry.readlines():
  plus, sha, message = line.split(" ", 2)
  for existing in lines:
    esha,emessage = existing.split(" ", 1)
    if message == emessage:
      break
  else:
    unmatched.append((sha, message))

for sha, message in unmatched:
  print "+ %s %s" % (sha, message)

Usage:

$ git checkout upstream
$ git log --oneline >all_commits.txt
$ git cherry -v upstream my_branch >cherry_commits.txt
$ python filter_cherry.py cherry_commits.txt all_commits.txt

SQLAlchemy 0.5.3 and 0.5.5 error: Set changed size during iteration

siker — Mon, 03 Aug 2009 02:52:15 +0000

Update 1: See Mike Bayer’s comment below – looks like there’s a fix for this coming up in 0.5.6. Very responsive work by Mr. Bayer. Thanks!

YippieMove has been running into some spurious problems with SQLAlchemy. When upgrading Python from 2.5 to 2.6 the errors became a lot more frequent. The error is “Set changed size during iteration” and happens in all kinds of random code paths when using the ORM layer. Today I saw it happen when getting a result object from a query by index, and then again when iterating over the result of another query.

The end of the traceback looks something like this:


  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/dynamic.py", line 192, in __iter__
    sess = self.__session()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/dynamic.py", line 181, in __session
    sess.flush()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/session.py", line 1354, in flush
    self._flush(objects)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/session.py", line 1359, in _flush
    if (not self.identity_map.check_modified() and
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/identity.py", line 56, in check_modified
    for state in self._mutable_attrs:
RuntimeError: Set changed size during iteration

The code in question looks like this:


    for state in self._mutable_attrs:
        if state.modified:
            return True

The program is single threaded and it sure didn’t look like “state.modified” would cause the iterated over set to change. Weird. After snooping around for a while I found this post which suggests an async GC being the problem.

Sure enough the problem appears to have gone away when automatic GC is disabled. It’s not a nice workaround because the garbage collector in Python collects cyclic garbage. But it did solve the problem for now so here it is if anyone else needs it:

import gc
gc.disable()

Unless you want to be buried in garbage, make sure to collect it once in a while in a safe spot in your code using gc.collect().

Hopefully this will be fixed in a future version.

Norwinter Studios » python

Find git commits which still need to be merged

SQLAlchemy 0.5.3 and 0.5.5 error: Set changed size during iteration