Kristian Klette

Migrate Django models to UUID primary key

Thu, 28 Sep 2017 14:17:40 +0000

Sometimes, old design decisions comes back to bite you. This is one of those tales.

A project I’m working on had a Django model similar to this:

class Municipality(models.Model):
    code = models.CharField(max_length=2, primary_key=True)
    name = models.CharField(max_length=100)

and it was used by other models such as:

class ZipCode(models.Model):
    code = models.CharField(max_length=2, primary_key=True)
    municipality = models.ForeignKey(Municipality)

And all was good, until we needed to expand the municipality model to support different countries, and thus a single unique field with only the code - which may collide across countries, was not enough.

For all the modern parts of the code base we use UUIDs as the primary key, so we wanted to migrate the primary key of Municipality to a UUID, while retaining all the relations.

As of September 2017, Django does not support migrating primary keys in any nice manner, so we’re on our own here.

We tried a lot of magic solutions, but we always got stuck with the migrations system not being able to detect and handle the changes nicely.

After some research and quite a bit of trial and error, we settled on the following approach. It has some draw backs I’ll get back to, but works reasonable well.

A quick reminder on how the world looks from a database perspective. When you define a ForeignKey-field in Django, it creates a database column of the same type as the primary key of the referenced model on the referring model, and adds the foreign key constraints. So in the example above, we have two tables (in pseudo SQL):

CREATE TABLE municipality (
   code varchar(2) PRIMARY KEY NOT NULL,
   name varchar(100)
);

CREATE TABLE zipcode (
   code varchar(2) PRIMARY KEY NOT NULL,
   municipality_id VARCHAR(2) REFERENCES(municipality.id) NOT NULL
);

So, we need to break the foreign key constraints, alter the root model, then map the the new primary key ids to the old ones, and re-apply the foreign keys to it.

We start by breaking the foreign keys in the referring models.

class ZipCode(models.Model):
    code = ...  # Same as before
    municipality = models.CharField(max_length=2)

python manage.py makemigrations -n break_zipcode_muni_foreignkey

Now that the Municipality model is free from any external referring models, we can go to work on it.

Start by adding the new id field:

class Municipality(models.Model):
    id = models.UUIDField(default=uuid.uuid4)

python manage.py makemigrations -n add_id_field_to_muni

For some reason, the default didn’t work in my case, so I added a step in the created migration to create new unique ids for all rows.

def create_ids(apps, schema_editor):
    Municipality = apps.get_model('loc', 'Municipality')
    for m in municipality:
        m.id = uuid.uuid4()
        m.save()
        
# ...

operations = [
    migrations.AddField(...),
    migrations.RunPython(code=create_ids),
]

Now we have a UUID id-field on Municipality, and we should be able to switch the primary key around.

class Municipality(models.Model):
    id = models.UUIDField(default=uuid.uuid4, primary_key=True)
    code = models.CharField(max_length=2, unique=True)

Create the migration, and make sure that the AlterField operation for code is run before the AlterField on id. We’ve added primary_key to the id field and unique=True to the code field, since we still want to enforce that constraint for now, and we lost it when we removed the primary_key attribute from it.

Congratulations, we now have a new UUID primary key. But we still need to clean up everything we broke the foreign keys from.

Lets start by creating an empty migration:

python manage.py makemigrations --empty -n fix_zipcode_fk_to_muni_uuid loc

Open the file, and let us begin:

def match(apps, schema_editor):
    ZipCode = apps.get_model('loc', 'ZipCode')
    Muni = apps.get_model('loc', 'Municipality')
    for zip_code in ZipCode.object.all():
        zip_code.temp_muni = Muni.get(code=z.municipality)
        zip_code.save()
    
# ...
operations = [
    migrations.AddField(
        model_name='zipcode',
        name='temp_muni',
        field=models.UUIDField(null=True),
    ),
    migrations.RunPython(code=match),
    migrations.RemoveField(model_name='zipcode', name='municipality'),
    migrations.RenameField(
        model_name='zipcode', old_name='temp_muni', new_name='municipality'),
    migrations.AlterField(
        model_name='zipcode',
        name='municipality',
        field=models.ForeignKey(
            on_delete=django.db.models.deletion.PROTECT,
            to='municipality')
]

Let us go through the steps here.

Add a temporary field for storing the UUID of Municipality that we want to connect to. We don’t make it a ForeignKey-field just yet, as Django gets confused about the naming later on.
We run the match function for looking up the new ids by the old lookup key, and store it in the temp field.
Remove the old municipality field
Rename the temp field to municipality
And finally migrate the type to a foreign key to create all the database constraints we need.

And there you go. All done.

There are some down sides here. Since we split the migrations into several files/migrations, we leave ourself vulnerable if any of the later migrations fail. This will probably leave the application in a pretty unworkable state. So make sure to test the migrations quite a bit. You can reduce the risk by hand editing all the steps into one migration, but if you have references from multiple different apps, then you need to the breaking and restoring in separate steps anyway.

Logging / Debugging

You’ll most likely end up with some SQL errors during the process of creating these, so a nice trick I like to do is to create a simple logging migration operation.

def log(message):
    def fake_op(apps, schema_editor):
        print(message)
    return fake_op
    
    
 # ...
 
 operations = [
     migration.RunPython(log('Step 1')),
     migration.AlterField(..),
     migration.RunPython(log('Step 2')),
     # ...
 ]

This allows you to see where in the process you fail.

To see what SQL Django creates for a given migration, run python manage.py sqlmigrate <appname> <migration_number> - this is super useful for checking whether operations are run in the order that you expect.

Easy steps for better living with Java

Mon, 08 Sep 2014 17:58:00 +0000

Here are some easy steps I try to do on every Java-project I’m working on. They are based upon what I’ve seen as major problem sources in different code bases, and should be fairly easy to apply - at least on new code.

Tooling

The first steps are to enable some basic tooling that can help us keep the project a bit of better shape. All of these comes with a cost, namely build time, but it’s totally worth it.

Enable strict compilation

The javac-compiler exposes flags that increases the amount of linting the compiler does, and how it should treat the warnings produced by the compiler.

Enabling it is as easy as adding -Xlint:all -Werror to the compiler arguments.

Using maven with the maven-compiler-plugin version 3.1 and up, simple do something like this in your pom.xml:

<build>
  <plugins>
    <plugin>
      <artifactId>maven-compiler-plugin</artifactId>
      <groupId>org.apache.maven.plugins</groupId>
      <version>3.1</version>
      <configuration> 
        <compilerArgs>
          <arg>-Werror</arg>
          <arg>-Xlint:all</arg>
        </compilerArgs>
        <encoding>UTF-8</encoding>
      </configuration>
    </plugin>
  </plugins>
</build>

or if you use Gradle, then this should do the trick:

compileJava {
   options.compilerArgs << "-Xlint:all" << "-Werror"
   options.encoding = 'UTF-8'
}

Use static analysis

Static analysis is an extremely useful tool for catching potential errors early. Aside from the built-ins in the compiler, my favorite tool is FindBugs. The latest release (3.0) includes support for Java 8, which has been a deal breaker for a while.

Adding it as a part of your build is quite easy, add the following snippet to your pom.xml:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>findbugs-maven-plugin</artifactId>
    <version>3.0.0</version>
    <configuration>
        <effort>Max</effort>
        <threshold>Low</threshold>
        <xmlOutput>true</xmlOutput>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
</plugin>

This does add a bit of time to your build, but I think it is worth it. If your codebase is so large that this adds an unacceptable amount of build time, consider running it on jenkins or something.

Don’t allow duplicate library versions

Any non-trivial java application that uses third-party libraries are prone to having multiple versions of libraries included in the classpath. Since the java classloading is not guaranteed to be deterministic, this can cause you immense headaches really fast.

If you use gradle with the java plugin, you can detect these problems out of the box. Just add the following to your build file:

configurations.all {
   resolutionStrategy {
       failOnVersionConflict()
   }
}

If you use Maven on the other hand, you probably want the enforcer plugin. To use it, add the following to your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-enforcer-plugin</artifactId>
    <version>1.3.1</version>
    <executions>
        <execution>
            <id>enforce</id>
            <configuration>
                <rules>
                    <DependencyConvergence />
                </rules>
            </configuration>
            <goals>
                <goal>enforce</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The code

These steps are more related to the code you write. There are, as with all things, exceptions where these guidelines does not apply, but in the general case, these should serve you well to remember.

Null is the exception

We’ve all seen NullPointerException stacktraces in our logs, and they are always annoying to fix. In my experience, they are often just fixed with a null-check, and then passing the problem on the caller of the function where occurred. This results in a code base littered with null-checks, and a lot of defensive - or blatantly unsafe code, where you are never sure if something is set or not.

Unfortunately we have no way of disabling nulls in Java, so all the burden is on us to defend the outer borders of our code against this. Now, some data is inherently optional, but they can be marked as such. In Java 8 you have a built-in Optional<T>-type, and if you are on 1.6 or 1.7 you can use the Optional-class from Guava.

So, instead of returning null, return Optional.empty() or if returning an empty value does not make sense, throw an exception. Its what they are for.

Do not allow broken data

Most enterprisy java projects have to deal with broken external systems, such as old databases, or some SOAP-service where everything is optional. Regardless of the type, we should guard our own software from the dirty data. Use Optional<T> for the fields where is does make sense, and if the data provided from the external system does not hold up to a minimal set of requirements, don’t pass it along, drop it on the floor!

If these causes some requirement not to be fulfilled, then fix it in the integration layer, but do not send shitty data down the line.

One way of ensuring that bad instances of your models can be created, use the constructor and validate the input.

public class Person {
   private final String name;
   private final Integer age;

   public Person(final String name, final Integer age) {
      this.name = checkNotNull(name, "name cannot be null");
      this.age = checkNotNull(age, "age cannot be null");
   }
}

checkNotNull(obj, message) is included with Guava, and throws a NullPointerException if the given instance is null.

One pattern we have deployed successfully is to have a “safe factory” for the model, which returns an Optional<T> of the instance.

public Optional<Person> safeCreate(final String name, final Integer age) {
   try {
    return new Person(name, age);
   } catch (NullPointerException npe) {
    LOGGER.warn("Invalid data provided for {}. Name: {}, Age: {}", Person.class.getName(), name, age);
    return Optional.empty();
   }
}

Using this pattern you can easily map input data to a list of valid instances. Using Java 8, one might do something like this:

externalSystem.getPeople().stream()
.map(p -> Person.safeCreate(p.getName(), p.getAge()))
.filter(Optional::isPresent)
.map(Optional::get)
.collect(Collectors.toList());

or in Java 6/7:

List<Optional<Person>> transformed = new ArrayList<>();
for (RawPerson rp : externalSystem.getPeople()) {
  transformed.add(Person.safeCreate(rp.getName(), rp.getAge()));
}
List<Person> persons = Optional.presentInstances(transformed);

The point is not to include objects that you are unsure of through the stack. The complexity starts dominating the system so fast.

Default to immutable objects

Prefer and default to Immutable objects. Throw out the old Java Beans book, and enter into a much nicer world. Creating mostly immutable objects is quite easy. Mark your fields as final, and only allow them to be set during construction. NO SETTERS!

Collections require a bit more attention, but using the immutable collections from Guava takes you a long way.

If you need to modify the instance, return a new copy with the modified data.

IoC Containers

IoC containers are quite popular, with Spring arguably is the most popular one. They can do you good, but be aware of the ease. Since it is so easy to inject instances all over the place, things get real messy very fast. See https://klette.us/some-tips-to-prevent-dependency-hell-in-the-land-of-ioc/ for patterns on how to reduce the pain.

Avoid if possible. There is nothing wrong with calling new MyClass() by yourself.

Reflection

Reflection is an escape-hatch that should only be used if you have no other options. It might be tempting, but it destroys everything from performance, to automated refactoring tools, and flow analysis tools.

Please just dont. Find another way, and you’ll be happier for it.

Documentation

In most real projects, there comes a time where you have to get your hands dirty, and code some complex logic, or deal with some other broken system. Documentation in these cases is paramount.

I have seen a secondary effect to documenting the code, that is documenting the methods before you write the implementation. Similar to writing the tests first, this seems to, for me at least, keep me much more honest and the methods are much clearer.

Give it a try for a couple of days.

Building Java 8 apps with Gradle

Sun, 29 Dec 2013 14:59:24 +0000

If you try to build your java-app with gradle against JDK8,and you error out on the tests being run, then this is most likely due to Gradle using ASM4.x for test-detection.

This has been fixed in Gradle 1.10 by using ASM5-beta to scan for tests, instead of 4.x as used in previous versions.

If you are unable to upgrade to Gradle 1.10 for some reason, you can apply the following trick to skip the scanning of tests, and just include them manually.

test.scanForTestClasses = false
test {
    include 'api/**/*Test.class'
    include 'service/**/Test*.class'
    exclude '**/*$*.class' // exclude anonymous inner classes
}

Keeping sane in the depth of madness

Sat, 28 Dec 2013 21:19:00 +0000

This last year I’ve been working on systems that integrate with a master ERP-system over SOAP. As with all thing enterprise, things tends to get a bit messy.

Most interactions with the SOAP-interface requires the user to specify a lot of identifactors for other objects. In our system, most of these are represented as Long-instances.

A naive implementation of this does not a happy programmer make.

A typical service for talking through the SOAP may look a bit like this.

public void triggerAccountAgreementUpdate(
	Long account,
    Long agreement,
    Long userTypeId,
    Long addressId,
    Long updateTypeId, ....) {}

On the surface, that might not be so bad, but give it some time, a couple of more parameters that suddenly are needed, and the typical call of this methods looks something like this:

srvc.triggerAccountAgreementUpdate(
    12313243L,
    14223421L,
    2L,
    234251L,
    1065L, ...)

This is starting to be hard to read. Some of the readability issues can be solved with declaring all the paramteres as properly named variables. One more problem that appears is when you have multiple ids of entities that look like each other. One example of this that I’ve found is where you have different types of addresses that you work with. If the type is only encoded in the names, these can be mistaken for each other, creating weird bugs that are hard to track down.

Given that this is in a statically typed (though with a bit broken type system) programming language, we should be able to leverage types to guide us.

In languages such as Scala and Go, this can easily be done via type aliasing

type CustomerId int32
type AddressId int32

In Java, we have no such luck, and must go through the usual boilerplate hell.

public final class CustomerId {
	public final Long id;
    public CustomerId(final Long id) {
    	this.id = checkNotNull(id);
    }
    // .equals and .hashCode below
}

By creating these types we can be a bit more explicit in our methods, and give the programmer a bit more information to work with.

public void triggerAccountAgreementUpdate(
	AccountId account,
    AgreementId agreement,
    UserTypeId userTypeId,
    AddressId addressId,
    UpdateType updateTypeId, ....) {}

The overhead of doing this might be discouraging, but we’ve had great results by using this technique. I think this becomes more and more important as the complexity of the application grows. Given a n-layered stack where the data has to flow through a wide range of methods and indirections before actually being used, then it is crucial not to mess this stuff up.

For more examples of this technique and others take a look at Jessica Kerrs excellent talk from Øredev this year.

Additional note

Since REST and JSON is all the rage these days, here is a small tip for using these identifier types using jackson.

Given the entity Customer defined like this:

public class Customer {
    public final CustomerId id;
    public final String name;
}

then the default serialization would produce something like this:

{"id": {"id": 1234}, "name": "Alice Example"}

Thats a bit silly.

We can fix this by using the @JsonUnwrapped-annotation provided by Jackson.

public class Customer {
    @JsonUnwrapped
    public final CustomerId id;
    public final String name;
}

If you use this annotation, then other annotations thats should affect the field, must be put in the CustomerId-class, not in the Customer-class.

For using the field as a *Param in Jersey (JAX-RS) the id-classes must obey the rules given in https://jersey.java.net/documentation/latest/jaxrs-resources.html#d0e1811. TL;DR of these are provide a constructor that takes a single string-argument for constructing the entity, or have a static method valueOf(string) that can create new instances.

Dependency hell in the land of IoC

Thu, 17 Oct 2013 11:54:53 +0000

Using automated dependency injection has made the life of a developer a bit easier, or so it seems for a while. Maybe a bit too easy. In this post I’ll describe some of the mess this can lead to, and what you should do to avoid it, or fix it if it has already happened.

The examples and mentioned libraries are Java-based, but most languages have something similar.

What is dependency injection?

Dependency injection (DI) is at its core a way of passing dependencies to a class to it by some means.

It can be easily explained by some piece of code:

class NoDependencyInjection {
    private OtherService os;
    private DifferentService ds;

    public NoDependencyInjection() {
        os = new OtherService();
        ds = new DifferentService();
    }
}
class WithDependencyInjection {
    private OtherService os;
    private DifferentService ds;

    public WithDependencyInjection(OtherService os, DifferentService ds) {
       this.os = os;
       this.ds = ds;
    }
}

The main difference is that the class NoDependencyInjection is responsible for creating and managing its dependencies on its own, while the WithDependencyInjection-class does not care how the dependencies are created, its just passed them, and uses their exposed API.

One common problem with the NoDependencyInjection-class way of doing things is about testability. Given that OtherService talks to a database, how do you make it so that you can properly unittest NoDependencyInjection?

You’d have to create some mock database, and do a lot of setup for OtherService, which NoDependencyInjection really should know nothing about, or create some sort of factory, that you can for each test specify the sort of object to return, something like ServiceFactory.setService(myServiceStub). The problem with the factory method is that your tests cannot run at the same time, since you are most likely using the same factory instance to create the service.

In my opinion OtherService should be testet on its own, and NoDependencyInjection should just rely on OtherService to function the way it is supposed to.

Once a program reaches a certain size and complexity, the dependency setup starts to get a bit cumbersome. This led to the arrival of DI-frameworks. The Spring-project might contain the most used one, and provides the developer with the @Autowire-annotation. This tells the framework to inject a ready made instance of that dependency into to the class at runtime. This annotation has two basic usage patterns, field annotation and constructor annotation.

class Service {
	@Autowired // Field annotation. Gets injected via reflection
    private OtherService os;
    
    @Autowired // Inject at construction
    public Service(OtherService os) {}
}

The problem

One problem that often occurs are classes that have way more dependencies than any class should have. Automated dependency injections makes it, perhaps, a bit too easy to add a new dependency to a class, without properly thinking about the consequences. This is even easier on classes that have no tests attached to them, since you never see the cost of setting the class up to a functional state.

These classes tend to look something like this:

class MyService {
    @Autowired
    private OtherService os;
    
    @Autowired
    private OtherService2 os2;
    
    @Autowired
    private OtherService3 os3;
    
    @Autowired
    private OtherService4 os4;
    
    @Autowired
    private OtherService5 os5;
    
    @Autowired
    private OtherServiceN osN;
    
    // .. methods are about five pages down

}

This is what I’d like to call legacy-by-design. It might work, but is nearly impossible to test (and if you manage it, it’s likely 500 lines of mock setup before you even get to test anything).

Why does this happen? For starters, It’s clearly not tested - the pain of testing this would have caused the developers to split the service long before that mess happened. Once a class gets, say 10 dependencies, the broken window effect starts to creep up on you.

Ugh, this is messy. Oh well, whats one more dependency? I’ll just add it.

Who cares about one more broken window in a house full of them?

So what can be done?

One small observation I’ve made is that if you use constructor based injection, it seems to deter people from adding more arguments more than adding a new field. My data set is quite small, so take this with a grain of salt, but the initial findings are promising.

So, if you use DI-frameworks, try to use constructor based injections. A nice bonus effect is that it is much easier to test, as you don’t have to bring up a Spring context in this instance, or something like Mockito’s @InjectMocks-magic. Still using Spring’s @Autowired annotation, this is easily done like this:

class MyService {
    private final OtherService os;
    
    @Autowired
    public MyService(final OtherService os) {
        this.os = os;
    }
}

This only gets you so far though, and does not really fix the classes that are already broken. To fix these, you’ll just have to roll your sleeves up and get ready for the treachorous road of refactoring untested code. The book Working Effectively with Legacy Code has some great techniques for doing such work. But the gist of the processes is finding natural groups of functionality that can be moved to its own class. If no such groups are really present, which sometimes happens in complex integration work, you can consider moving some of the methods used from the dependencies to a new interface, with as few methods as possible, and create a facade class that proxies the calls to the dependencies. This allows you to stub the interface for tests more easily, and new patterns are likely to emerge.

Let’s see if we can create a decent example.

Given this class:

@Service
class BookingService {
    @Autowired
    private RoutingService routingService;
    
    @Autowired
    private AvailabilityService availabilityService;
    
    @Autowired
    private LocationRepository locationRepository;
    
    @Autowired
    private CustomerRepository customerRepository;
    
    @Autowired
    private StaffRepository staffRepository;
    
    // .. Imagine a lot more dependencies here
    
    public Booking requestBooking(
        Long customerId,
        WorkType workType,
        DateTime dateOfWork
    ) {
        Customer c = customerRepository.get(customerId);
        Location l = locationRepository.get(c.homeAddress());
        Set<Staff> possibleStaff = staffRepository.findForWorkType(workType);
        
        Set<Staff> staffInRangeAndAvail = Sets.newHashSet();
        for (Staff staffMember : possibleStaff) {
            Long distance = routingService
                                .distanceBetween(l, staffMember.officeLocation());
            if (distance < 10000 &&
                    availabilityService.isFree(staffMember, dateOfWork)) {
                staffInRangeAndAvail.add(staffMember);
            }
        }
        
        if (staffInRange.isEmpty()) {
            return null;
        }
        
        return new Booking(staffInRange.get(0)); // Something like that        
    }
}

So, whats going on here. A quick glance shows that the requestBooking method uses five dependencies, which is quite a lot.

One possible route is to encapsulate the things that have to do with finding the staff members that have the right skills and are in range into a seperate class.

So, we can create a quick interface for the functionality

interface WorkorderStaffFinder {
    Set<Staff> staffForWork(WorkType type, Location l, DateTime timeOfWork);
}

and a quick implementation

class DefaultWorkorderStaffFinder implements WorkorderStaffFinder {

    private RoutingService routingService;
    private AvailabilityService availabilityService;
    private StaffRepository staffRepository;
    
    @Autowired
    public DefaultWorkorderStaffFinder(...) {} // assign dependencies
    
    Set<Staff> staffForWork(
        WorkType type,
        Location l,
        DateTime timeOfWork
    ) {
        Set<Staff> possibleStaff = staffRepository.findForWorkType(workType);
        Set<Staff> staffInRangeAndAvail = Sets.newHashSet();
        for (Staff staffMember : possibleStaff) {
            Long distance = routingService
                            .distanceBetween(l, staffMember.officeLocation());
            if (distance < 10000 &&
                    availabilityService.isFree(staffMember, dateOfWork)) {
                staffInRangeAndAvail.add(staffMember);
            }
        }
    }
}

The original class now looks like this:

@Service
class BookingService {
    
    @Autowired
    private LocationRepository locationRepository;
    
    @Autowired
    private CustomerRepository customerRepository;
    
    @Autowired
    private WorkorderStaffFinder workorderStaffFinder;

    
    // .. Imagine a lot more dependencies here
    
    public Booking requestBooking(
        Long customerId,
        WorkType workType,
        DateTime dateOfWork
    ) {
        Customer c = customerRepository.get(customerId);
        Location l = locationRepository.get(c.homeAddress());
        Set<Staff> staff = workorderStaffFinder
                            .staffForWork(workType, l, dateOfWork);
                            
        if (staffInRange.isEmpty()) {
            return null;
        }
        return new Booking(staffInRange.get(0)); // Something like that        
    }
}

We have removed two dependencies, and we can now more easily test the business logic of requestBooking.

If you think about it, why should the requestBooking-method care how the possible staff members are found? It should only care if some are found or not. This is one aspect of the single responsibility principle that is a part of SOLID (or some other buzzy-abbreviation).

Ideally we should end up with a BookingService-class that we can test like this:

@Test
public requestBookingShouldReturnNullIfNoStaffIsAvailable() {
    BookingService bs = new BookingService(
    						customerRepositoryMock,
    						locationRepositoryMock,
                            new EmptyResultWorkorderStaffFinder());
    
    Booking b = BookingService.requestBooking(1L, WorkType.TECH, DateTime.now());
    
    assertThat(b).isNull();
}

class EmptyResultWorkorderStaffFinder implements WorkorderStaffFinder {
    Set<Staff> staffForWork(...) {
        return new Set<Staff>();
    }
}

Java dependencies

Thu, 10 Nov 2011 23:08:00 +0000

A little python snippet for creating dot-representation of the dependencies in a java app.

Let this little baby chew on your target/classes dir and you will get a dot-file on stdout.

 #!/usr/bin/env python

import re
import os
import sys
from os.path import join, basename

LIB_REG_EXP = re.compile('[,+L#][a-z][A-Za-z0-9/]+')
PACKAGE_FILTER = ''
CLASS_PREFIX_REMOVE = ''

if len(sys.argv) > 2:
    PACKAGE_FILTER = sys.argv[2]

if len(sys.argv) > 3:
    CLASS_PREFIX_REMOVE = sys.argv[3]

def get_class_dependecies(class_file_contents):
    libraries = set(map(lambda x: x.replace('/','.'), [l[1:] for l in LIB_REG_EXP.findall(class_file_contents)]))
    return filter(lambda x: PACKAGE_FILTER in x, libraries)

class_dependecies = {}
for root, dirs, files in os.walk(sys.argv[1]):
    if '.svn' in dirs:
        dirs.remove('.svn')
    class_files = [f for f in files if f.endswith('.class') and '$' not in f]
    for class_file in class_files:
        class_content = open(join(root, class_file)).read()
        class_name = join(root, class_file).replace('/','.')[len(CLASS_PREFIX_REMOVE):-6]
        if PACKAGE_FILTER not in class_name:
            continue
        class_dependecies[class_name] = []
        for dep in get_class_dependecies(class_content):
            if dep == class_name:
                continue
            class_dependecies[class_name].append(dep)

print "digraph G {"
for key, deps in class_dependecies.items():
    for dep in deps:
        print '"%s" -> "%s";' % (key, dep)
print "}"

The second argument creates a basic filter for the class-names you want. The third parameter is used for removing the folder-names (target.classes. in usual cases)

Example:

python dotclass.py target/classes us.klette.myproject target.classes. > deps.dot

Have fun

Rot

Sat, 08 Oct 2011 12:30:19 +0000

I’m quite fascinated by how quickly code rots and becomes a mess, despite the programmers best intentions. A lot could be fixed by testing and and refactoring more often, but it always seems to come second to getting shit done, and does sometimes have unintended effects.

For some reason I see this more often in Java code than in other languages. Maybe it’s because of its class-madness creating deep nested code (try to trace a call through Jersey, I dare you.).

Jersey is actually a nice example of this. The code is in itself clean, documented and well tested, but it is still (in my opinion) a mess. Seems to me that abstraction does not only have an performance penalty, but a readability issue as well. And in this case, it seems the code has been tested and refactored multiple times, but still it’s hard to understand.

Seems like there is a fine line between reusability and loss of understanding for what happens inside the black box.

Photo by Will Sculling

Take control over your data: banking

Mon, 18 Jul 2011 07:46:00 +0000

So, banking data is quite interesting. If one could get a hold of your credit card transactions in an easy way, that would be sweet. But, its not that easy.

My bank, DNBNor, has a feature that you can download either an excel document, or a CSV formatted dump of the transactions against one of your accounts on a monthly basis. The programmers choice would be CSV, but I found the CSV exports to be spotty, so excel it it.

But I don’t want to my analysis in Excel, so Perl to the rescue. The following snippet takes an Excel document, exported from DNBNor, as it’s first argument, and gives you an JSON-representation of the data to STDOUT. Should be quite easy to manipulate these in any language of your choice.

#!/usr/bin/perl

use strict;
use utf8;
use warnings;
use Switch;
use Spreadsheet::ParseExcel;
use JSON::XS;

my $excel = Spreadsheet::ParseExcel->new;

die "Need an xls-file containing DNB Nor transaction list as frowst argument\n" unless @ARGV;

my $doc = $excel->Parse($ARGV[0]) or die('Could not parse excel file\n', $@);

my @transactions = ();

for(my $sheet=0; $sheet < $doc->{SheetCount} ; $sheet++) {
    my $current_sheet = $doc->{Worksheet}[$sheet];
    for(my $row = $current_sheet->{MinRow}; defined $current_sheet->{MaxRow} && $row <= $current_sheet->{MaxRow}; $row++) {
        # Skip header row
        if ($row == $current_sheet->{MinRow}){
            next;
        }
        my %transaction = ();
        for(my $column = $current_sheet->{MinCol}; defined $current_sheet->{MaxCol} && $column <= $current_sheet->{MaxCol} ; $column++) {
            my $cell = $current_sheet->{Cells}[$row][$column];
            if (defined($cell)){
                switch ($column){
                    case 0 { $transaction{'date'} = $cell->Value || undef; }
                    case 1 { $transaction{'description'} = $cell->Value || undef; }
                    case 2 { $transaction{'interest_date'} = $cell->Value || undef; }
                    case 3 {
                        $transaction{'debit'} = $cell->Value || undef;
                        $transaction{'credit'} = '0.00';
                        }
                    case 4 {
                        $transaction{'credit'} = $cell->Value || undef;
                        $transaction{'debit'} = '0.00';
                        }
                }
            }
        }
        push @transactions, \%transaction;
    }
}

my $json = JSON::XS->new->utf8->encode({'transactions' => \@transactions});
print STDOUT $json . "\n";

exit 0;

GIST available at https://gist.github.com/ea0c6ef9e712e2caec88

Requires Spreadsheet::Excel and JSON::XS - both available at CPAN.

Have fun :-)

Word cloud of my thesis so far

Wed, 15 Jun 2011 17:11:29 +0000

Word cloud of my thesis so far. I guess you might be able to guess what I’m writing about :-)

apache2-mpm-itk and libapache2-mod-wsgi

Thu, 31 Mar 2011 10:58:00 +0000

If you are running Apache 2 with mpm-itk and mod_wsgi on a debian system, you’re a bit out of luck performance wise.

From the MPM-ITK documentation:

mpm-itk is based on the traditional prefork MPM, which means it’s non-threaded; in short, this means you can run non-thread-aware code (like many PHP extensions) without problems. On the other hand, you lose out to any performance benefit you’d get with threads, of course; you’d have to decide for yourself if that’s worth it or not. You will also take an additional performance hit over prefork, since there’s an extra fork per request.

So mod_wsgi in embedded mode will make apache load your whole python project each request. That seems quite inefficient, so we’ll run mod_wsgi’s daemon mode.

mod_wsgi’s daemon mode launces background deamons that caches your code, and even have have built-in support for running under mpm-itk. The only problem is that for this support to be enables mod_wsgi must be compiled against an mpm-itk patched apache 2. In Debian this is not the case, so the daemon sockets will be owned by www-data and not the user that you assigned to the vhost.

You will see something like this in your apache error log:

[Fri Mar 18 13:11:40 2011] [error] [client 2001:700:300:9::188] (13)Permission denied: mod_wsgi (pid=22237): Unable to connect to WSGI daemon process 'myuser' on '/var/run/wsgi/wsgi.22197.0.1.sock' after multiple attempts.,

I’ve filed a bug with debian on this isses (#619252), but no responses yet.

Luckely it’s not very hard to build your own mod_wsgi package to support daemon mode under mpm-itk.

Here is a quick how-to on the matter:

$ apt-get source libapache2-mod-wsgi
$ cd mod-wsgi-3.3
$ mkdir debian/patches

$ cat - > debian/patches/series
fix-mod-wsgi-for-itk.patch
^D

$cat - > debian/patches/fix-mod-wsgi-for-itk.patch
Remove #ifdefined for mpm-itk support, and only support mpm-itk, as we only use mpm-itk
--- a/mod_wsgi.c
+++ b/mod_wsgi.c
@@ -10037,11 +10037,7 @@
*/

 if (!geteuid()) {
-#if defined(MPM_ITK)
 if (chown(process->socket, process-&gte;uid, -1) < 0) {
-#else
-if (chown(process->socket, ap_unixd_config.user_id, -1) < 0) {
-#endif
 ap_log_error(APLOG_MARK, WSGI_LOG_ALERT(errno), wsgi_server,
"mod_wsgi (pid=%d): Couldn't change owner of unix "
"domain socket '%s'.", getpid(),
^D

$ debchange -i
$ dpkg-buildpackage

the resulting package will work as expected. I haven’t tested if it works on any non-mpm-itk enabled apache, but I wouldn’t bet on it.

Now, go forth and hack some python web apps!