One way to do Django on AppEngine

June 3rd, 2011 § Comments Off on One way to do Django on AppEngine § permalink

Disclaimer: I work for Google.  Any and all opinions expressed or interpreted below are mine own and not that of my employer.

I’ve built dozens webapps in the past 10 years (thats right, i was writing web apps at 13) using all sorts of tools.  It started with raw HTML and spaghetti code perl.  Then it moved onto spaghetti code PHP.  Then it was PHP in a framework.  Then it was asp .NET and finally for the past 3 years or so I’ve used exclusively Django.

So naturally when a friend of mine approached me with an idea for a project using Google’s new prediction API I immediately thought to use Django.

But what’s the fun of doing a project the same way you’ve been doing it for 3 years?  Django + Apache + EC2 is boring.  I figured, I work for Google, perhaps I should be using a Google hosted solution.  It was time to use AppEngine.

Full stop.  Think for a minute.  Django, a highly relational tool built on databases like Mysql and Postgresql, seems fundamentally at odds with the philosophy of AppEngine — non relational, highly distributed.  Fudge!

OK, breathe.  There has to be a way to make this work.  From some research I discovered:

  • You *CAN* use Django + MySQL in AppEngine.  The most common technique I uncovered involves hosting MySQL outside of AppEngine and establishing a secure connection from AppEngine to the remote DB.  This seemed a bit excessive and wasteful — the whole point of this exercise is to use AppEngine!
  • Or, you could use Django-Nonrel

Django-Nonrel is a version of Django that modifies the Django query generator to work ontop of a non-relational database.  Beware though, this has some inherent drawbacks.  Complex joins and inheritance, for obvious reasons, no longer work.  Before continuing any further read through the entire django-nonrel page and all the limitations and make sure you can live with them.  I was able to, you probably (with a bit of work) can too.

That doesn’t solve all our problems though.  We still don’t have a connection between the Django application and the AppEngine datastore.  Thats where DjangoAppEngine comes in.  DAE provides the following, incredibly awesome, capabilities:

  • A connection between the django-nonrel query system and the AppEngine datastore.
  • Automagically sets up AppEngine logging for your Django App.
  • Automagically sets up the Django URL handling system within the AppEngine app — all you need to do is setup your!
  • Most importantly, provides new hooks within to allow you Django Shell access to your production AppEngine datastore!  While not critical once your app is deployed, for development and debugging this capability is invaluable!

So, brave adventurer, be off with you!  Explore the wonderful world of building applications with the ever popular python ORM Django ontop of the ever scalable AppEngine platform!   My experience was slow and painful, and involved a lot of head scratching and googling, but in the end all was working.   Read the documentation carefully and bliss shall be yours too!

Link to my app coming soon…

    Trying out the Nexus S

    December 16th, 2010 § 2 comments § permalink

    I got a Nexus S today, and here are my initial thoughts, when comparing it to my Nokia N900.

    • No LED indicator light.  This is driving me nuts.  It’s rather jarring to not be able to look at my device and know whether or not it needs my attention.
    • It’s FAST!  Opening applications is instantaneous, scrolling is smooth, no slow downs whatsoever after the first day.
    • Android multitasking is rather annoying coming from Maemo.  I’m not in love with how it works, its kind’ve like “every app is always open” and to switch back to an application you just reopen it.  Thats nice, but it doesn’t lead to any nice workflows with great multitasking.
    • There is a lot more application support in the android ecosystem, which is nice coming from Maemo.
    • The lack of a physical keyboard is going to take some getting used to.  I’m not particularly thrilled about the on screen keyboard quiet yet.
    • Installing applications is quick and painless.  Much less frustrating than on Maemo.
    • I can actually use the email client!  It has good search features!  It doesn’t take 20 minutes to load!  Oh happy day!
    • There are… ads….?  In my applications?  This is weird.  I don’t like it.
    • SUPER integration with all Google services, which is sort’ve a big deal for me now a days.

    The Google services integration is actually the deal breaker for me, and is probably what will keep me on Android going forward.

    Update: So I’ve been using the phone for a couple of days now.   Here are some more thoughts:

    • Battery life is ‘meh’.  I felt like i had a lot more control over the battery usage on the N900 — changing from 2G/3G, disconnecting from the network etc. where all much simpler operations on the N900.
    • Using the phone, just for day to day stuff, is a pleasure.  By far the most ‘fun’ to use where it ‘just works’ a very large percentage of the time.
    • Voice commands are simply *phenominal*.  “Send text to daniel Hi dan, want to meet for drinks?” actually works about 90% of the time!  Or  “Goto” or “navigate to pizza” all work rather well.  It’s what voice recognition should be like.
    • The navigation app is slow and almost unusable.  Nuts.

    Introducing the Personal *nix Cluster Manager

    November 29th, 2010 § 7 comments § permalink

    I was at home this holiday, with the family enjoying the Thanksgiving Holiday.  I was messing around with stuff on my netbook and realized I wanted files from my desktop back in my apartment.   Only problem was — I have no idea what my IP is back there (I moved recently).  I’m also too lazy to setup my own or pay for a proper dyndns solution.  Why is it that in 2010 I still have this kind’ve problem?  Shouldn’t there be some application I can run on all my client pcs which keep them meshed together?

    A bit of digging, couldn’t find anything.  So I wrote one.  Introducing the “Personal Cluster Manager for *nix/ssh systems”.

    First, an important assumption: This software is meant to be used by you and you only.  It probably won’t scale to 1000 machines and I’m perfectly fine with that.  I have a handful (10?) machines I care about keeping track of for my stuff and I imagine most of the people who are interested in this are on the same order of magnitude.

    Ok, enough nonsense, what is it?  There are three pieces:

    1. A client daemon that maintains a reverse ssh tunnel with a master, always connected and accessible node.  Theres a good bit of configuration to get this right, so theres also a little helper script to ask the user questions to get the right config values for the client.
    2. A “master” script.  This provides  an interface to a sqlite database which keeps track of what hosts we’ve seen, what ports their tunnels are on etc.  You can ask the master “show me all hosts you know about” or “which tunnels are active and usable right now?”.
    3. A curses based UI.  The UI pulls all its data from the master program, and doesn’t do any direct querying.  (This is intentional, as I intend to build other kinds of UIs and want all the data stuff to be handled underneath the UI).  The UI shows you what tunnels are active to which hosts, and with one keypress can give you a shell on any of the remote machines (using the reverse tunnel setup by the client).

    All 3 of the above are working in version 0.1 in github.

    A very poor quality screencast with typos and minor bugs left and right, but you’ll get the idea: (nevermind the ‘scrollz’ stuff, thats something particular about my local bashrc and nothing to do with PCM)

    AMD Joins Meego with Nokia and… Intel?

    November 15th, 2010 § 2 comments § permalink

    Source Press Release

    Wow.  My first thought was “What, AMD and Intel working together?”  My second thought was “Awesome!  More people behind Meego!”   My third thought was “Wait, AMD is supporting a software platform?  That’s new.”

    One at a time:

    First — Intel and AMD both understand that they are in the fight for their lives right now, but necessarily just with each other, but with ARM.  ARM via Android and iPhone is taking over more and more market share in the global processor market.  All of our smartphones are running ARM, most of your tablets are running ARM and ARM is moving more and more into the netbook space.  A quick Google for “ARM vs X86” turns up not technical comparisons of the two ISAs but instead recent article after recent article of market analysis between the different chips.  It therefore makes sense for both of them to get into the game and promote some cross-platform software.

    Second — I honestly believe right now that smartphone OS’s are a two horse race, iPhone vs. Android.  RIM and WP7 will fight it out for a small bit of enterprise market, but the lions share is the two goliaths.  Meego could very well be a solid third contender, which I think is awesome for many reasons.

    • More competition means consumers win.
    • Meego is also my personal favorite.  Debian in your pocket?  Sweet.  Meego promotes free and open source linux-desktop technologies whicih benefit not just the Meego platform but Linux desktops everywhere.  That’s right — every dollar that Intel, Nokia and now AMD sink into Meego goes into a technology that, more often than not, will be integrated into the next version of your Ubuntu or Fedora desktop.  Examples?  Telepathy, dBus, Tracker, gstreamer, webkit,  gtk, qt, clutter, kernel power management, low power drivers, apt, rpm and the list goes on.

    Thirdly, and what I think is by far the most interesting, is the possibility that this is the begining of a new era for AMD.  AMD for the longest time was heads down on designing and fabricating chips.  About 18 months ago AMD spun off the fabricating part of that equation into a company named Global Foundaries.  Since then AMD has been chip and system design through and through.  AMD’s chief competitor though, Intel, has for the longest time been a big hardware and software shop.  This strategy has worked rather well for Intel — promoting software which promotes their Intel chips.  Things like Moblin, the Intel C compiler etc. all contribute here.  AMD could really take advantage of this idea and run with it, running with what I am going to call the “Google Trick”.   There is probably a much better term for this, but what I mean is to support products and services which might not be direct revenue drivers in order to support complimentary products which do drive revenue.  Google I think is king at this — products like Picasa, Wave, Google Voice, Goog 411, Google Sketchup, Google Earth, etc. etc rarely make a dime, yet they all contribute to “Googliness” and eventually drive Google search, which is key.


    Full Disclosure:  I own AMD stock.

    A Must Watch: Barack Obama’s Post-Election Press Conference

    November 3rd, 2010 § Comments Off on A Must Watch: Barack Obama’s Post-Election Press Conference § permalink

    I do recommend downloading the mp4 and watching at 2-3x speed though… a bit slow otherwise.

    Some great questions and responses from Barack:

    • What I feel is accurately describing WHY america is so frustrated with the current political situation
      • “In such a hurry to get things don’t we didn’t get a chance to reform HOW we get things done”
    • Working on economy, job creation
    • Describing how to reasonably go forward with respect to dissenting opinions on healthcare
    • “Talk specifics” — which previsions do we like and don’t like and how can we compromise to make the most progress and do the best for our country (as opposed to meaningless ‘lets just repeal this’ type conversation).

    The People Bug Tracker:

    October 24th, 2010 § Comments Off on The People Bug Tracker: § permalink

    I mentioned a few posts back that would come around, and after a lot of procrastination it is finally here!

    First things first — a couple thank yous and relavent links are in order:

    • dreamhost, for making the one-click-install so utterly painless
    • edgewall for providing trac as open and free software
    • The guy at 119labs for writing this blog post which detailed very nicely how to install the trac openid plugins inside a dreamhost one-click-install.  This is actually non-trivial because without dreamhost PS you do not get root or the ability to add eggs to the system, so you need to setup your own local python virtualenv to build the needed eggs.  There is also some config kung-fu that needs to be done to make the SQL schemas happy.

    So, what is this bug tracker about?  We’re not concerned about software here, this is a “Bug Tracker for People.”  Did your roomate forget to buy milk when he finished the last carton?  Spouse always leaving the toilet seat up?  Not cleaning up the sink?  These are common everyday problems that we nag each other about and can actually lead to real frustration and interpersonal drama.  Finally there is an avenue to enable documented discussion about these issues and come to civil and mutually beneficial conclusions.

    The “lawsuit club”

    October 9th, 2010 § Comments Off on The “lawsuit club” § permalink

    There are so many major corporations suing each other over complete drivel now a days that I’ve lost track of who is pointing the lawsuit-press-generator at who. Below is an aggregate from a few minutes of Googlin’.  Kudos for the idea for the title goes to the NYTimes.

    (Note: this is still a work in progress)

    • Motorola Sues Apple
      • When: 10/6/2010
      • Type: Patent Infringement
      • Number of Patents: 18
      • Patent Claims Cover: Signal Protocols, 802.11 (wifi), antennas, wireless email, proximity sensing, software application management, location-based services and multi-device synchronization
      • Hilarious Company Claim: “After Apple’s late entry into the telecommunications market, we engaged in lengthy negotiations, but Apple has refused to take a license. We had no choice but to file these complaints to halt Apple’s continued infringement.”
      • Responses:
      • Outcome:
      • Links:

    Don’t use third party tools

    September 22nd, 2010 § Comments Off on Don’t use third party tools § permalink

    That is, unless you plan on owning them and supporting them.

    Allow me to explain. There are some tools, like the Linux Kernel or the Python Interpreter that you’ll be able to use (for the most part…) without a second thought. (Though, there are definitely imaginable, and in fact scenarios I’ve run into, where tweaking and investigation of these tools is required).

    No, for the most part I want you to be wary of niche tools. Things like the Django framework, the Zenoss monitoring suite, random python libraries, system maintenance tools etc. Don’t get me wrong, these are fantastic pieces of software which are each excellent in their own right. But if you chose to use them in your business or in a project consider whether or not you’re willing to maintain the tool as well; could you get away with something inhouse that is much smaller, with many fewer features, but is tailored to exactly your needs?

    Why? Well these tools each have use cases in mind. I bet dollars to donuts your specific use case will align with the use case of the tool, but not quiet perfectly. You’ll eventually do somethng ever so slightly out of scope. The tool will seem like it’ll work fine, but soon you’ll find that it is not perfect and you’ll have to hack it in some way, change a few lines of code here and there, add some extra script to support an ancillary function, run a cron job to periodically clean logs etc.

    And that is fine. It is expected that you’ll need to do some tool sharpening.

    But you probably won’t realize how much time you’ll sink into these tools. Some examples in my recent history: The Zenoss Monitoring Suite. A fantastic tool for monitoring cloud infrastructure. But what happens when you’re scaling your busienss at 200% or more per quarter? Zenoss worked fine with 20 machines, but how does it handle 200, or 2000? Ordinarily sure Zenoss can do 2000 machines, but we started adding custom templates to the nodes to collect extra data tailored to our application. Pretty quickly we found Zenoss either running out of system resources (we’ve moved it to larger boxes several times), or falling behind entirely or flat our having bugs which cause a certain use case to not work (Maintenance windows don’t quiet work as well as one would hope…). We’ve spent all in all hundreds of developer hours just working out ways to make Zenoss work for us. At the end of the day we’re finding that many of our original use cases can actually be handled perfectly fine with independent monitoring scripts entirely separate and outside of Zenoss; avoiding a lot of headache overhead. I can come up with at least a dozen other examples for other tools I’ve used including but not limited to the Linux Kernel itself, the python interpreter (who hasn’t had GIL or memory problems?), Django, Django-Piston, the Hudson Continuous Integration Environment, the Graphite graphing/monitoring tool, puppet, iclassify, nfs, nis, etc.

    So, support your tools. Contribute your changes back upstream to help out the next person who happens to run into your use case. Just don’t try and support more than you can.

    I’ll end this rant on one final point. Obviously you can only really make use of tools you can support, and you can only support tools you have the source and capability to build. Another great reason to never use proprietary software!

    Building Empire State Buildings

    September 7th, 2010 § Comments Off on Building Empire State Buildings § permalink

    Today Google had one of its fancy playable logos. They really are nifty – I actually found myself playing with it and admiring the physics for a couple of minutes. And then I noticed…. So was everybody else. I got a couple of messages from friends who also spent a couple of minutes with it. That got me thinking… How much productivity does the world lose when Google does this?

    To make life simpler, let’s just think about the USA.

    According to Internet World Stats there are 230 Million internet users in the USA alone. Lets be conservative and say 10% of them saw the logo today and spent an average of 2 minutes playing with it. (though, given that every user probably makes at least one search during their online session and Google has a 70%+ market share in search thats probably very very conservative).

    23 Million people each spending 2 minutes = 46 Million Minutes = 767,000 Man-Hours = 31900 Days = 87 man-years of human productivity.

    According to all of Wikipedia took roughly 100 Million man hours to create. So, Google’s fancy logo consumed about as much time as creating 1% of wikipedia, in the United States. Given that the US actually only accounts for a moderate amount of total internet traffic (and assuming Google used the fancy logo in every country) that number could really be 5 or 10%.

    The Empire State building was built in 7 Million man hours, so in the US alone we could’ve put up 10.2 floors (its 102 stories) of the Empire State building!

    Now, remember the pacman logo? I definitely spent more than 2 minutes on that… combine both logos and you have a whole Empire State Building 🙂

    How to VNC into any Linux machine (or how I ‘hack into’ my friends computers)

    September 6th, 2010 § 1 comment § permalink

    The biggest challenge in helping out a friend in need (running Linux, of course — I don’t bother troubleshooting any Windows problems with any tool other than an Ubuntu Live CD now-a-days) is often their local networking setup. Why? Because I can point out tutorials or guides until I am blue in the face but some people are just very uncomfortable reconfiguring their system and insist they will ‘mess it up’, so sometimes you have to just do it help fix things yourself and firewalls get in the way.

    The quick thing to do is SSH into their machine, or VNC in, and just troubleshoot the problem yourself. However, 9 times out of 10 a router, or a NAT, or a firewall, or a proxy, or an act of god will prevent you from establishing a TCP connecton to the machine. To get around this problem most commercial troubleshooting applications require the user to initiate the connection. So, I figure, why can’t we use some quick open source tunnel magic to do the same?

    Here is my quick and handy guide to “hacking in” (aka (reverse) ssh port forwarding) to somebody’s PC that is behind a firewall or NAT:

    0) You need yourself a machine which has an open ssh port (22 by default) to the internet for this to work (herein referred to as It doesn’t have to be your machine, it can be a third party machine that you have a login to. All it needs is to be running openssh on it. A node, or a dreamhost shell account, anything like these will do. I personally use a cluster I still have access to at my undergraduate college the University of Penn (yay [email protected]!).

    You’ll also need a login on <YOUROPENPC>.com (herein: for your friend (or they can borrow yours) as well as a login on your friend’s machine (duh) and preferrably also root on your friend’s machine (if only temporary).

    1) When you first help your friend setup their computer add an extra desktop shortcut for them which will run the following command (or walk them through typing in this command)

    ssh -C -R 8888:localhost:22 [email protected]<YOUROPENPC>.com

    When your friend runs this command it will prompt for a password. Now you can either give them their own username and password on the open-to-the-world machine or you can temporarily change your password (if you dont mind giving them access to your account… assuming they have no idea what a terminal is I find this option works fine for most cases).

    So, what does this do? “-R” means “Reverse Port Foward” — When SSH makes the connection to the “” server it sets up an extra port on “” (port 8888) to automagically redirect back to port 22 on localhost. Therefore any data sent to port 8888 on actually is rerouted over the original ssh connection to port 22 on localhost. (localhost in this case means your friend’s computer).

    (-C just means enable compression. If you’re going to do VNC forwarding (see below) this will help keep bandwidth usage down and speed up)

    2) Next, you need to login to

    ssh [email protected]

    3) From here, you can now do:

    ssh -p 8888 [email protected]

    and Voila! You’re now logged into your friend’s computer despite any firewall or NAT they might have!

    Alright, but what if you REALLY need access to their X Session (aka what they see on their screen) to help with the issue? We can fix that too.

    4) Install x11vnc and run it on your friend’s PC (which you can now do because you’re logged into their machine)

    sudo apt-get install x11vnc && x11vnc -display :0

    5) x11vnc will by default host a vnc session on port 5900. We, however, only have access to port 22 on our friend’s computer (remeber the :22 in the initial ssh -R command?). Nothing is stoping us from running another ssh session with a new port number though! So, from your friend’s PC, log into again!

    ssh -C -R 8889:localhost:5900 [email protected]

    Now we have two different ssh connections from your friend’s computer to One sets up port 8888 to redirect back to your friend’s computer on port 22 (for ssh connections) and one redirects back to your friends computer on port 5900 (for VNC). You could change this to any port you want — 80 for HTTP if your friend has a webserver, 631 if you want to access their CUPS printing page etc.

    6) We’re not quiet done, however. Only port 8889 can talk to the x11vnc server on your friend’s machine. Unless you have an X session with a vncviewer on you still can’t connect. So, what we now need to do, is connect a port on YOUR machine to port 8889 on This is a “forward” port forward and can be done quiet simply.

    ssh -C -L 9999:localhost:8889 [email protected]

    Now any data sent to port 9999 on your machine will actually be routed via ssh to port 8889 on Any data sent to port 8889 on actually is routed via another ssh session to port 5900 on your friend’s computer. Ah, what a tangled web we weave!

    7) We’re almost done! Now, all we need to do is setup the vnc session by pointing to localhost port 9999 (which redirects to which redirects to your friend’s computer)

    vncviewer localhost:9999


    Recap of what we covered:

    • ssh -R — Reverse port forwarding. This allows you to setup a port on the REMOTE machine which points to your LOCAL machine.
    • ssh -L — Normal port forwarding. This allows you to setup a port on the LOCAL machine which points to the REMOTE machine.
    • ssh -C — Enable compression in your ssh connections.
    • Port 22 — what ssh uses by defualt
    • Port 5900 — what VNC uses by default
    • Port 631 — what CUPS (Linux/Unix printing services) uses by default
    • x11vnc — a program which starts a VNC session based on an already running X session
    • vncviewer — a standard program to connect to a VNC session.