Norwinter Studios » Programming

Diary app Chronicle.im peeks out.

siker — Sat, 17 Jul 2010 20:38:32 +0000

Our latest project, Chronicle.im, which we’ve been working on secretly for about nine months now, now has a teaser preview up online at www.chronicle.im. In the next few weeks we hope to start opening up for beta testing of this online journal writer, so if you want in now is the last chance to sign up. Just enter your email address on that page. I guarantee we wouldn’t mind if you sent us a tweet or two regarding the site too!

Find git commits which still need to be merged

siker — Sat, 29 May 2010 02:14:35 +0000

If you have a git branch which you are working on which is a clone from an upstream ‘official’ branch, and you sometimes submit your commits to be integrated upstream, there’ll surely be times when your patches are not accepted or fall between the cracks. After a while it can get a little hairy to figure out which commits are unique to your branch and which ones are incorporated in the main repo.

The ‘git cherry’ command can help with this. It lists all commits in one branch which are not in another. It works like this simply:

git cherry -v upstream my_branch

The man page describes the command succinctly:

Every commit that doesn’t exist in the branch has its id (sha1) reported, prefixed by a symbol. The ones that have equivalent change already in the branch are prefixed with a minus (-) sign, and those that only exist in the branch are prefixed with a plus (+) symbol.

The problem is that sometimes the cherry command does not identify ‘equivalent changes’. I’m not sure why, but perhaps this happens when the main tree has cherry picked commits and made changes to whitespace or resolved merge conflicts. Or maybe it doesn’t work well at all with cherry pick. Either way, I had a list of changes which cherry returned which I knew had in fact been merged. So I whipped up up this silly little Python script to filter a cherry results list from a complete log of the upstream repo and to remove all commits with the same commit message. This is obviously not bullet proof as different commits may have the same message. But it was good enough for me in this case. So without further ado, here it is:

import sys

cherry = open(sys.argv[1], "r")
filtr = open(sys.argv[2], "r")
lines = filtr.readlines()
unmatched = []

for line in cherry.readlines():
  plus, sha, message = line.split(" ", 2)
  for existing in lines:
    esha,emessage = existing.split(" ", 1)
    if message == emessage:
      break
  else:
    unmatched.append((sha, message))

for sha, message in unmatched:
  print "+ %s %s" % (sha, message)

Usage:

$ git checkout upstream
$ git log --oneline >all_commits.txt
$ git cherry -v upstream my_branch >cherry_commits.txt
$ python filter_cherry.py cherry_commits.txt all_commits.txt

Good Django Celery Overview

siker — Wed, 14 Oct 2009 17:18:41 +0000

Happy stream of thoughts has a good overview over how RabbitMQ, Celery and Django work together to make distributed task processing easy and convenient in Django. The example described is for video processing.

RabbitMQ, Celery and Django

Migrate Eventum to Redmine.

siker — Wed, 26 Aug 2009 23:22:29 +0000

I have migrated more stuff to the new Redmine setup. This time I pulled all issue data from an old and well used Eventum setup. I made some educated guesses about how the Eventum data was laid out and then hacked up the Redmine trac migration script until it worked for Eventum.

It’s really not a very pretty script but since I won’t be doing any more work on it I’ll release it here in the spirit of open source. Whether it works for you or not is anyone’s guess, but one thing is for sure: if it was just sitting on my hard drive it definitely would not.

Download: migrate_from_eventum.rake

See instructions in the beginning of the file.

Refactor Serializable Java Class

siker — Thu, 20 Aug 2009 21:56:20 +0000

Today I renamed and moved a class in a Java program. It was a Serializable class containing some preferences. As we are getting rid of it I renamed it from Preferences to LegacyPreferences, with the intention of starting a new Preferences class using a regular config file instead of serialization. Bad idea. Java refused to load serialized data into the new class for two reasons: one obvious and expected and one seriously braindamaged in true Java fashion.

The first problem was that the deserialization class could no longer find the class to deserialize to. No wonder since it was in a new package and had a new name. No problem. I subclassed ObjectInputStream and overrode resolveClass(ObjectStreamClass desc) to properly load the class from its new name and location.

That’s when Java freaked out with the following piece of brilliance:

local class name incompatible with stream class name X

Well, duh. Of course the name is different, that’s the point of a refactoring. What exactly were they trying to accomplish when they added this name check? The class has already been found, it’s interface is an exact match. The name of the class is irrelevant at this point.

I proceeded by overriding readClassDescriptor() with the intention to simply replace the name of the class with the new name when reading the stream. Java would never even realize the class used to have another name. But no: that method must return a java.io.ObjectStreamClass object, which you can’t instantiate or override unless you’re in the java.io package. And if you just grab the super.readClassDescriptor() instance you aren’t allowed to make any changes to it.

At this point I considered my favorite solution to Java damage. I’d like to call it “corrective Java surgery” – use reflection to force your way into the class and fix it. This is a very gratifying solution since you can undo the mistakes in Java and set things right. The drawback of course is that different versions of Java may require different corrective code.

That wasn’t necessary in this case since reading the source code of ObjectInputStream revealed a static method for creating ObjectStreamClasses: ObjectStreamClass.lookup. With that tool things fell into place very simply like so:

@Override
protected java.io.ObjectStreamClass readClassDescriptor() throws IOException ,ClassNotFoundException {
	ObjectStreamClass desc = super.readClassDescriptor();
	if (desc.getName().equals("com.trueship.base.Preferences")) {
		return ObjectStreamClass.lookup(LegacyPreferences.class);
	}
	return desc;
};

Hope that’ll help anyone else looking to rename and deserialize classes in Java.

SQLAlchemy 0.5.3 and 0.5.5 error: Set changed size during iteration

siker — Mon, 03 Aug 2009 02:52:15 +0000

Update 1: See Mike Bayer’s comment below – looks like there’s a fix for this coming up in 0.5.6. Very responsive work by Mr. Bayer. Thanks!

YippieMove has been running into some spurious problems with SQLAlchemy. When upgrading Python from 2.5 to 2.6 the errors became a lot more frequent. The error is “Set changed size during iteration” and happens in all kinds of random code paths when using the ORM layer. Today I saw it happen when getting a result object from a query by index, and then again when iterating over the result of another query.

The end of the traceback looks something like this:


  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/dynamic.py", line 192, in __iter__
    sess = self.__session()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/dynamic.py", line 181, in __session
    sess.flush()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/session.py", line 1354, in flush
    self._flush(objects)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/session.py", line 1359, in _flush
    if (not self.identity_map.check_modified() and
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/orm/identity.py", line 56, in check_modified
    for state in self._mutable_attrs:
RuntimeError: Set changed size during iteration

The code in question looks like this:


    for state in self._mutable_attrs:
        if state.modified:
            return True

The program is single threaded and it sure didn’t look like “state.modified” would cause the iterated over set to change. Weird. After snooping around for a while I found this post which suggests an async GC being the problem.

Sure enough the problem appears to have gone away when automatic GC is disabled. It’s not a nice workaround because the garbage collector in Python collects cyclic garbage. But it did solve the problem for now so here it is if anyone else needs it:

import gc
gc.disable()

Unless you want to be buried in garbage, make sure to collect it once in a while in a safe spot in your code using gc.collect().

Hopefully this will be fixed in a future version.

Migrate MoinMoinWiki to Redmine.

siker — Sun, 02 Aug 2009 04:18:59 +0000

Today I switched a small wiki from MoinMoin to a new Redmine setup. I couldn’t find any existing scripts to convert the wiki pages over. For a moment I thought I could do MoinMoin to Trac and Trac to Redmine but I thought better of it. It’d be a lot of work and the wiki syntax would probably be completely mangled in the process.

Instead I hacked up Carl Nygard’s MediaWiki to Redmine script. It’s an awful hack really but it worked for the pages I needed, and it does preserve revision history.

Download the script: migrate_from_moinmoin.rake
Edit the source and enter your own email address instead of admin@example.com.
Place the file in lib/tasks/ in your Redmine installation.

Run the command with

rake redmine:migrate_from_moinmoin RAILS_ENV="production"

Enter a project identifier.
Enter an absolute path to the data/pages folder of your MoinMoin install.

Problem with jQuery 1.3 and :eq in a complex selector

siker — Fri, 31 Jul 2009 22:55:19 +0000

While trying to recreate this XPath as a jQuery selector I ran into some unexpected difficulties:

/html/body/table[3]/tbody/tr/td/table/tbody/tr

My first attempt to convert it looked something like this:

$("body > table:eq(2) > tbody > tr > td > table > tbody > tr")

This gave no results. I simplified it a little bit,

$("body > table:eq(2) tr tr")

Still no luck. After experimenting for a while it looked to me like :eq wasn’t playing well in this expression. The following worked:

$("body > table").eq(2).find('tr tr')

Weird.

Improved Cygwin Terminal

siker — Thu, 30 Jul 2009 17:51:38 +0000

The standard terminal in Cygwin is not very good. It can’t be made wider, you can’t select text normally using the mouse without first accessing the context menu and selecting ‘mark’, the default font is thick and ugly, etc. When you are used to the Mac OS X Terminal the Cygwin console is not fun.

Here’s the solution: puttycyg. It’s a modification of putty which allows it to act as a cygwin terminal in addition to the normal ssh and telnet functionality. Very cool!

Found it through this blog post. Thank you Andreas Hochsteger!

Pipe into the clipboard on OS X: pbcopy

siker — Mon, 27 Jul 2009 17:03:36 +0000

I just discovered this handy command in OS X: pbcopy. It lets you pipe the output of a command right into the clipboard (or the ‘pasteboard’) so you can paste it into some non terminal application next. A convenient way to avoid having to scroll up and select the text in the terminal by hand.

Example:

cat myfile.txt | pbcopy