You don’t want to test all of your views in a huge application by appending ?profile in the urls. At least I don’t. And even if you do, you might detect a view that takes few seconds to response, but it is called only twice a year. While another view taking a millisecond is called million times a day. Therefore the following decorator and a shell script can help you identifying such views.

The decorator stores function’s name, time it spent and its timestamp in a temporary file. The shell script then parses that file and displays sum of time used in each function. All you have to do now is add @execution_time in front of every function you’d like to measure.

class execution_time(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        from datetime import datetime
        n1 = datetime.now()
        result = self.func(*args, **kwargs)
        n2 = datetime.now()
        delta = n2 - n1
        file = open('/tmp/exectime', 'a')
        file.write('%d %s %d %d %d %d %d %d %dn' % (
            delta.microseconds, self.func.__name__,
            n1.year, n1.month, n1.day, n1.hour, n1.minute,
            n1.second, n1.microsecond)
        )
        file.close()
        return result
    def __repr__(self):
        """Return the function's docstring."""
        return self.func.__doc__

The shell script. Cat, (g)awk, and sort. Make a call with the file produced by the decorator as an input parameter.

#!/bin/bash
echo '  absolute    relative     #calls   time/call function'

cat $1 |
gawk '{ all += $1; a[$2] += $1; freq[$2] += 1; }
END { for(x in a) printf "%10d %11g %10d %11g %sn",
    a[x], a[x]/all, freq[x], a[x]/freq[x], x; }' |
sort -n

Finally, the output. Ordered by absolute time spent in function. The second column shows the percentage of time spent in a function, e.g., the model_preview function taking 10,61% of all measured time. Note that this is relative to the sum of time that was spent only by measured views. One of the drawbacks of this decorator is that it is not taking into account the fact that one function might call the other. So if function A calls B, and you measure both, you’ll end with time for A that is actually time for A and B together. But that’s when the profiling using middleware comes handy.

  absolute    relative     #calls   time/call function
    211772 0.000133614         21     10084.4 load
    368835  0.00023271         72     5122.71 sort_boxes
    384175 0.000242389         33     11641.7 remove_from_cart

...

 139603047   0.0880802       4585     30447.8 design
 160016266     0.10096       7074     22620.3 render_image
 168310848    0.106193       4172       40343 model_preview

You can use one of the python’s module named hotshot, high performance logging profiler. How to use it you can read on its presentation site. Here I will only mention that its statistics function (hotshot.stats) outputs results to the standard output.

Let’s decide we will trigger the profiler on any url and output its results in the browser by appending profile attribute in the url query string. In order to intercept that attribute we create a so-called view middleware. In this middleware we inspect the url that was used to call a certain view.

Now, in middleware code we just have to check whether there is a GET attribute named profile in the query string (e.g. http://example.com/some/path/?profile). If there is one we create the Profile object, use it to make a call to the actual view, and then output statistics via returning HttpResponse. Since statistics are output to stdout we redirect it to StringIO to be able to display it in the response.

If no such attribute exists None is returned, and normal procedure of calling the default view takes place. When you put it all together the middleware code looks like the following:


from django.http import HttpResponse
import hotshot, hotshot.stats
import sys, StringIO, os

class ProfileMiddleware():
  def __init__(self):
    pass

  def process_view(self, request, view, *args, **kwargs):
    for item in request.META['QUERY_STRING'].split('&'):
      if item.split('=')[0] == 'profile': # profile in query string

        # catch the output, must happen before stats object is created
        # see https://bugs.launchpad.net/webpy/+bug/133080 for the details
        std_old, std_new = sys.stdout, StringIO.StringIO()
        sys.stdout = std_new

        # now let's do some profiling
        tmpfile = '/tmp/%s' % request.COOKIES['sessionid']
        prof = hotshot.Profile(tmpfile)

        # make a call to the actual view function with the given arguments
        response = prof.runcall(view, request, *args[0], *args[1])
        prof.close()

        # and then statistical reporting
        stats = hotshot.stats.load(tmpfile)
        stats.strip_dirs()
        stats.sort_stats('time')

        # do the output
        stats.print_stats(1.0)

        # restore default output
        sys.stdout = std_old

        # delete file
        os.remove(tmpfile)

        return HttpResponse('<pre\>%s</pre>' % std_new.getvalue())

    return None

Next, you save this code in the file named middleware/profile.py. In order for middleware to work one must enable it in the settings.py file:


MIDDLEWARE_CLASSES = (
  ...
  'djangocode.middleware.profile.ProfileMiddleware',
)


The sample output might look like this:

19743 function calls (19238 primitive calls) in 0.064 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     4465    0.007    0.000    0.007    0.000 encoding.py:37(force_unicode)
      995    0.006    0.000    0.010    0.000 html.py:30(escape)
        8    0.005    0.001    0.041    0.005 defaulttags.py:108(render)
      995    0.004    0.000    0.030    0.000 debug.py:85(render)
     1006    0.004    0.000    0.014    0.000 functional.py:246(wrapper)
     32/1    0.004    0.000    0.049    0.049 __init__.py:764(render)
     1027    0.004    0.000    0.004    0.000 safestring.py:89(mark_safe)
     1010    0.003    0.000    0.005    0.000 __init__.py:690(_resolve_lookup)
     1010    0.002    0.000    0.002    0.000 context.py:38(__getitem__)
     1009    0.002    0.000    0.009    0.000 __init__.py:533(resolve)
     1020    0.002    0.000    0.007    0.000 __init__.py:672(resolve)
     42/3    0.001    0.000    0.005    0.002 __init__.py:254(parse)
        1    0.001    0.001    0.064    0.064 pages.py:76(account_statistics)
      324    0.001    0.000    0.001    0.000 util.py:39(__getattr__)
     1405    0.001    0.000    0.001    0.000 __init__.py:790(render)
        1    0.001    0.001    0.004    0.004 pages.py:53(_sql)
      317    0.001    0.000    0.001    0.000 cursors.py:320(fetchone)
      191    0.001    0.000    0.002    0.000 debug.py:25(create_token)
       37    0.001    0.000    0.001    0.000 __init__.py:487(__init__)
        3    0.001    0.000    0.002    0.001 debug.py:10(tokenize)
      191    0.001    0.000    0.001    0.000 __init__.py:229(create_token)
       10    0.001    0.000    0.001    0.000 cursors.py:273(_do_query)
   293/76    0.001    0.000    0.001    0.000 __init__.py:750(get_nodes_by_type)
      290    0.001    0.000    0.001    0.000 connections.py:189(string_decoder)

Note you should only install the profiling middleware in the development version since you don’t want just anybody to see the structure of your code from live version. And the next time you would like to check why a certain url is running slow just append it ?profile.

To get an answer I downloaded some GDP and population data from the world bank website and overall medal standings from the Beijing 2008 official website. After some data alignment a plot was created using R, the environment to statistically explore data sets. Each point of the graph represents one country. Blue circles represent countries whose athletes won at least one medal in Beijing. Their size correspond to the number of medals won. Red crosses represent countries not winning any medals. The position of the points (red and blue) on X axis represents log of its population number, while Y axis plots log of GDP per capita.

The first pattern that appears on the figure is the upper right triangle containing big blue points – representing countries winning the most of the medals. So GDP per capita matters. So does population size. Also left side and bottom of figure representing countries too small or too poor to win any of the medals.

Interesting are countries from former communist block with not so much wealth, but with great tradition in sports and population big enough to also win slightly bigger chunk of medals – such countries are Ukraine, Belarus, Romania, Kazakhstan, Poland, etc.

Countries from Africa mostly populate area in the lower part of the figure, where three examples stand out: Kenya, Zimbabwe, and Ethiopia. I believe mostly they won athletic competitions where they’re known for mastering long distance running.

What follows is the R code used to create this graph, where country, pop, gdppc, all, and color represent countries’ name, its population, its gdp per capita, sum of medals (gold + silver + bronze), and color representing whether the country won any medals at all (blue) or none (red):

postscript(file='beijing.ps')
plot(log(pop), log(gdppc), cex=all, col=color, pch=16)
text(log(pop), log(gdppc), labels=country, pos=4)
dev.off()


Conclusions? The rich can definitely play, and size matters as well.

I’ve put the k-nearest neighbours algorithm on work. The algorithm finds the nearest k samples to the given one (the one we’d like to predict), and then tries to assume its class/value somehow. So there are three tasks to make it work:

• define distance that will tell the algorithm which are the closest samples
• set k – neighbourhood size
• choose combination of the k values from the closest samples to obtain the final results

For a (user, movie) tuple I found the k closest users that watched the movie, and combined their rates to get a prediction.

Distance

There are many different distance measures to consider. One can try pearson’s correlation coefficient. Vectors’ sparsity brought cosine similarity to my mind. It works fine for clustering documents represented by sparse vectors. But none of those produced satisfactory results, so I tried another method that tries to capture some common sense.

So imagine a person walking into the room full of unknown people. Initially everybody has the same (some average) distance to you. Then you talk to them and the first one likes the same music band as you. So distance to this person diminished a little bit. Then another person says the same band is crap, so his/her distance increases. But there are more bands, movies, actors, moral issues, work related issues etc. you talk about. The more ‘slots’ you have in common and if you agree on each slot above some average agreement, the closer you are to the person.

So that’s exactly what I used to calculate the distance between users. First, i calculated the average distance between all the rates. Actually i took around one million randomly picked rates, and got an average rate-to-rate distance to be 1.53465. When comparing two users their distance was sum of the r1 – r2 – average_distance for the movies in common. And when the movie was not seen by one or both of them, average distance was assumed. That’s how the problem of non-overlapping movies was solved, when users had only a few movies in common.

K

With the distance set, initially i took 100 neighbours, and was a bit lucky, because later i tried also with 25, 50, 75, 125, 150, and 200 but none improved the score. Note also that for other distance the number is not necessarily the same.

Combination

I tried only two of them. one is averaging the 100 rates obtained from the closest samples, and the other is weighted, where weights were dropping linearly from 100 for the closest sample toward 1 for the farthest one. The later gave better results.

Implementation

Built two c++ classes, namely user and movie with all the necessary in their interfaces, such as user::get_closest_users(). Initially everything was written in c, but it was a messy solution. Actually one month later I couldn’t read the code anymore, I got lost in the arrays and pointers…

48 cores attacking netflix prize

Running the qualifying dataset took me about five days. Some friends did some caching of the neighbourhood, but didn’t work well. Then one day i got lucky. At work they asked me if i have some software to do some heavy weight testing of the 7 brand new hp blades (each armed with eight cores) to be tested. That’s 56 cores, heaven. Using python script I split the dataset in small pieces and distributed them around the blades and later picked up the results. Using ssh i managed to run it remotely. It took 4 to 6 hours to finish.

Wishes

I had more time to play with it.

The title should actually be tight coupling visualized. Graphs come from one of the companies where I used to work. Each node in the graph represents one function from the code (not necessarily in the same file). An edge running from the node a to the node b represents a call of function b from function a.

How to obtain such a graph from your code? Ok, first of all, it was PHP code. Function calls were obtained and saved in a file using the tool named PHPCallGraph. In this case I used it from command line on Ubuntu. After that it is suggested to use GraphViz, but it was too slow and graphics not nice. So i converted the output to a .gml format, and visualized it with another must-have companion – yEd. It’s java-based, it’s free, it can handle large graphs, and it has powerful arsenal of algorithms to get the desired layout. The graphs were obtained using the organic layout (check out the tool’s screenshot gallery on its website).

Now again about code and the business. One comes from a local project and the other from registration and billing system written using Jan Rain’s OpenID Library. Guess which is which.