Category: Projects

Introducing the Personal *nix Cluster Manager

I was at home this holiday, with the family enjoying the Thanksgiving Holiday.  I was messing around with stuff on my netbook and realized I wanted files from my desktop back in my apartment.   Only problem was — I have no idea what my IP is back there (I moved recently).  I’m also too lazy to setup my own or pay for a proper dyndns solution.  Why is it that in 2010 I still have this kind’ve problem?  Shouldn’t there be some application I can run on all my client pcs which keep them meshed together?

A bit of digging, couldn’t find anything.  So I wrote one.  Introducing the “Personal Cluster Manager for *nix/ssh systems”.

First, an important assumption: This software is meant to be used by you and you only.  It probably won’t scale to 1000 machines and I’m perfectly fine with that.  I have a handful (10?) machines I care about keeping track of for my stuff and I imagine most of the people who are interested in this are on the same order of magnitude.

Ok, enough nonsense, what is it?  There are three pieces:

  1. A client daemon that maintains a reverse ssh tunnel with a master, always connected and accessible node.  Theres a good bit of configuration to get this right, so theres also a little helper script to ask the user questions to get the right config values for the client.
  2. A “master” script.  This provides  an interface to a sqlite database which keeps track of what hosts we’ve seen, what ports their tunnels are on etc.  You can ask the master “show me all hosts you know about” or “which tunnels are active and usable right now?”.
  3. A curses based UI.  The UI pulls all its data from the master program, and doesn’t do any direct querying.  (This is intentional, as I intend to build other kinds of UIs and want all the data stuff to be handled underneath the UI).  The UI shows you what tunnels are active to which hosts, and with one keypress can give you a shell on any of the remote machines (using the reverse tunnel setup by the client).

All 3 of the above are working in version 0.1 in github.

A very poor quality screencast with typos and minor bugs left and right, but you’ll get the idea: (nevermind the ‘scrollz’ stuff, thats something particular about my local bashrc and nothing to do with PCM)

A UPnP Home, Step 2

Part 1.  Ordering a hardware UPnP-AV Renderer

The Vera I ordered in “A UPnP Home” was to act as a bridge between UPnP and appliances around the house.  This still left my AV needs somewhat unfulfilled as most UPnP capable devices are UPnP consumers, but do not offer themselves up as fully fledged renderers (AVTransports).  For example, the PS3 can find my Rygel mediaserver and play content from it but I cannot control the PS3 with Zhaan.  Only the PS3 can control the PS3 (lame).

I’ve been looking for a real UPnP mediarenderer for a while now, and finally pulled the trigger on one the other day.  The most common device in this market is the “Popcorn Hour 100”.  There is another version, called the eGreat EG-M34a which is what I purchased   Its the same hardware just in a different package.  It calls itself a “Network Media Tank”, which seemed a bit odd but sure, I’ll go wit hit.  It promised to be controllable remotely via UPnP and also boasts support for every format known to man.

Part 2.  Testing out the Network Media Tank

I plugged in the NMT with an HDMI cable to my TV, plugged in the ethernet and power, switched to HDMI input and…. nada.  I pulled open my handy dandy GUPnP universal control point and low and behold the NMT shows up on the network.   I told it to start playing some music and whoosh, my TV came to life.  (I have to wonder how other people get started with this thing, those without their own control points).  Once the thing turned on it did eventually show me a UI on the TV for browsing my media shares.

So thats the good news, I can start playing music via UPnP.  The eGreat NMT does the most basic thing we need from a renderer, cool.  But how does it fare in responding to the rest of the UPnP spec?  Answer: really, really poorly.  Here are a list of problems I have found so far which inhibit this NMT from being a “good” UPnP renderer

  • The “Pause” action is mysteriously missing from the UPnP device description and the NMT does not respond when a Pause is sent.  Pausing via the remote works fine.  I guess I could hack around this by noting the current time when the user hits pause, send a Stop(), then when the user hits Resume I could hit Play and then Seek.  That is really lame though.
  • The GetPositionInfo() command always returns a CurrentURIMetaData with Title, Artist and Album being “Unknown”.
  • GetPositionInfo()  always returns a song length of 00:00 making it impossible to draw a progress bar.
  • There is no RenderingContorl service so no UPnP volume control.

The good news is these are all very easy issues to fix in software.  My hunch is this whole thing is Linux based, so I am hoping the source code will turn up someplace and I’ll be able to hack in these features and load up some custom firmware.

Part 3.  Testing the Vera / Home appliance automation

The vera was an interesting case.  You can tell this is a startup — plain white box with no branding, the box itself is all white and the Vera logo is clearly a sticker somebody put on by hand.  I’m not one to nitpick based on aethestics though, so I plugged it in and booted it up.  My experience:

  • The instructions say to try and go to 192.168.81.1 which didn’t work
  • I found, via UPnP multicast, that the device was 192.168.1.111 (I could’ve also asked my router’s dhcp table to find it).
  • Going to the device via the browser (at 1.111) worked perfectly.  I got nice totally unhelpful video tutorials.
  • Most of the online documentation for Z-Wave device additions is old and outdated.  I guessed at how to add a new z-wave device (by holding the z-wave button on the vera and on the new device) and got lucky and the appliance showed up.
  • GUPnP control point noticed a new device under the Vera device, with a SetTarget() function.
  • SetTarget() was able to successfully enable and disable my lights!

From here I will probably build some kind of Maemo application which will either just change Scenes in the vera (the vera is capable of storing appliance configuration in prepackaged scenes which can be switched between) or I might even just build a UI around SetTarget()).  Either way — Vera works perfectly (after some rough edges).

Conclusion So Far:

The jury is still out on finding a good, cheap UPnP-AV Media Renderer.  The eGreat is OK, but has some serious UPnP software deficiencies.  The Vera looks like it will solve all of my UPnP/Appliance needs though.  Overall I am happy for now, and am excited to hack together some Maemo apps to work with the Vera and to try and find software workarounds via Zhaan to make my eGreat more usable.

p.s. I had a pretty terrible sinus headache when writing this article, so I apologize if my prose is less exciting and more grammatically terrible than usual.

PyGI 0.5.1 Released

Hello,

About 3 weeks have passed since the initial release of PyGI.  Since
then we have seen a significant bump in activity in #pygi and also
several new contributors.  A big thanks to John (J5) Palmieri for his
continued efforts to test PyGI with his applications and drive
development forward, as well as Tomeu Vizoso for his hard work and
John Dahlin for his continued support and patch reviews.

Major pieces of this release include the refactoring of the invoke
method (which needed it badly) as well as new overrides, a bunch of
callback support fixes and union support.  Shortlog is attached below
for more.  Python 3 support is still in the works and should land
soon.

PyGi 0.5.1 : http://ftp.gnome.org/Public/gnome/sources/pygi/0.5/pygi-0.5.1.tar.gz

You can find the PyGi website/wiki:

http://live.gnome.org/PyGI

And the public git repo at:

git://git.gnome.org/pygi

Bug tracking is done in project pygi on bugzilla.gnome.org.

Cheers,
-Zach

John (J5) Palmieri (4):
make __all__ be a list of strings, fix override mechanism to use
it correctly
Add the Gtk.Builder override
override Gdk.Drawable to add cairo_create convinience method
wrap GObject module so we can go through GI when requesting attrs

Simon van der Linden (1):
Bump required GLib version to 2.22

Tomeu Vizoso (15):
Add support for enums without GType
Rename variable with a very generic name
Print any error messages raised inside _pygi_closure_handle
Refactor get_* methods in the *Info wrappers
Add basic support for unions
Dont force subclasses to implement all virtual methods of their bases
One more step at refactoring _wrap_g_function_info_invoke
Add override for Gdk.Color
Fix GAsyncReadyCallback
Move invocation code to its own file
Fix passing GDestroyNotify
Only hookup vfunc implementations for locally-defined methods
Avoid freeing garbage
Fix passing callbacks as constructor args
Fix overrides.Gdk.Color.__new__ args

Zach Goldberg (7):
Bump version during development to 0.5.1
Refactor implementation of scope call to allow for multiple
calls during lifetime of function invocation.
Change SCOPE_TYPE_INVALID handling to be a warning and not an error
Force out arguments to be initialized as NULL.  Comes with a test.
Change SCOPE_TYPE_INVALID handling to be a more verbose error.
Correct the reference counting of userdata in closure handling
Step 1 of refactoring _wrap_g_function_info_invoke

A UPnP Home

Since Zhaan made it into extras-devel I have gotten several feature requests for new functionality.  Many of them have made sense and I have gone ahead implemented them in the latest version.  For example, Zhaan 0.1-8 implements vastly improved playlist support.  Zhaan is now at the point where I have found myself actually using it enough to entirely replace my standard Sonos controller and actually ENJOY the experience.  (I also installed Zhaan on my ‘stable’ N900, not just my development device, and have witnessed no ill effects.  I think it’s probably safe for anybody to use at this point)

Since (in my opinion anyway) I have succeeded at building an application to solve in-home media problems with UPnP I now turn to other problems that UPnP can solve (since it’s so much fun to hack around with).  Zeeshan Ali wrote a GUPnP-light application which shows an image of a lightbulb which goes “on” and “off” based on another test control point application. That seems like an interesting idea, using UPnP to control my lights.  And why not, right?  Who really wants to get out of bed at night to turn off the wall-mount light switch when my phone is on my night stand within arms reach and could be used to control those lights.  (clappers be damned).

Herein lies a rather common problem in computer science.  Writing software to do fun things in simulations is easy.  Applying it to solve real world problems by designing, manufacturing, marketing and selling a real physical product is hard.  Thus for the first 2 days of my search I could not find any devices to allow me to control my lights via UPnP.

During the first two days of searching I did learn a lot.  Here is a quick summary/FAQ of how modern “Smart Homes” work: (Smart meaning networked and remotely controllable)

  • Smart devices in the home do not often connect to the WLAN, let alone expose themselves as UPnP devices.
  • Instead, they use mesh networking.
  • There are several common mesh network protocols.   Z-Wave, X10, Insteon to name a few.
  • Once you deploy a bunch of Z-Wave (or other) mesh devices you need to deploy a “base station” of sorts which can control all of the devices and which MAY connect/bridge to the LAN.
  • Most base stations come with their own software to interact with via a browser or other proprietary mechanism.

Last night at about 3AM, on the verge of giving up my search, I found micasaverde.com.  They have a device, called the Vera, which looks to do exactly what I want!  I posted a quick question on their forums confirming that the device does what I had hoped (exposing each Z-Wave device as its own UPnP discoverable device). Turns out it does!   Here is a response from Javier, an “administrator” of the micasaverde forums.

Yep, that’s the main point.

Every device attached to Vera gets a full UPnP description and identification. Even for internal programming, everything is done using UPnP references.

In fact, writing device plugins (to get new devices attached to Vera) is mostly writing Lua code to join UPnP actions and state variables with the real-world device control interface.

To boot it looks like Vera is low power and designed for energy savings.  It also runs on a (mostly, from what I can tell) open source Linux stack.  Rock on!

Looks like I’ll be getting myself a Vera for my birthday (next week) and, with any luck, a week or two later posting another application to Maemo extras-devel that can control Z-Wave devices via a Vera!

Update: Turns out you can SSH into your Vera too.  I’m starting to build a collection of embedded Linux/OpenSSH servers around my house 🙂 (Chumby, Vera, N900 etc.)

Zhaan now available in extras-devel!

Myself and the PyMaemo folks have been working these past few days to clear the blockers from making PyGi work on maemo.   There was a last minute bug in the Maemo autobuilder which prevented PyGi from properly being imported into the maemo extras repository.  (Turns out the bug was caused by the fact that the PyGi package’s version number had capital letters in it.  Thanks X-Fade for your help!).  The bug has now been cleared and all of PyGi and Zhaan have been imported successfully.

For any users: Zhaan should be installable in the multimedia category of your application manager.  Please give installing and using it a shot and report any bugs you find (in the maemo.org BugZilla)!  Positive feedback and/or feature ideas is also appreciated!

PyGi version 0.5, the ‘Watch out, theres a Volcano!’ release

(Special note for this version of the release statement on this blog — All the features Zhaan needed made it into the release, hence Zhaan is 100% compatible with PyGi 0.5)

It is my pleasure to bring to the world the first release of PyGi –
the Python GObject Introspection bindings – version 0.5. We chose the
number 0.5 to indicate that we believe PyGi to be of beta quality. We
want to encourage the brave souls of the world to try it out and help
us by reporting any bugs that are found.

PyGi is nearly feature complete. A highly condensed list of features:

– Implements in and out arguments of all types
– Implements closures, callbacks and virtual functions
– Implements wrapping of structs, objects and interfaces.

A huge thank you is due to several people for making this happen.

– Simon van der Linden, for all of his hard work on PyBank/PyGi.
– Tomeu Vizoso for all of his hard work on PyGi throughout the past
months/years
– John (J5) Palmieri, Colin Walters, Johan Dahlin and others for
assistance, contributions and help along the way,
– All of the great folks who participated in the Gnome/Python 2010
hackfest who made this release possible (including generous support
from Red Hat, Canonical and OLPC)

Version 0.6 of PyGI is likely to include:
– Python 3.1 support (already 95% working! Thanks David Malcom and
John Ehresman!)
– A major refactoring of some of the thickest parts of the code
– Much improved documentation
– Improved compatibility with older bindings

PyGi 0.5 is available for download here:

http://ftp.gnome.org/Public/gnome/sources/pygi/0.5/pygi-0.5.tar.gz

You can find the PyGi website/wiki:

http://live.gnome.org/PyGI

And the public git repo at:

git://git.gnome.org/pygi

Bug tracking is done in project pygi on bugzilla.gnome.org.

Cheers,
-Zach Goldberg

PyGi Maintainers:
Simon van der Linden
Tomeu Vizoso
Zach Goldberg

Git Shortlog:

Alex Dedul (1):
Search for python-config-${VERSION} when python${VERSION}-config
is not found

Anderson Lizardo (1):
Depend on GLib 2.20 rather than 2.22

Colin Walters (2):
[Makefile.am] Clean up CFLAGS handling, don’t override all: target
Add Tomeu’s prototype script for converting pygtk to pygi

Johan Dahlin (3):
Create overridden modules in two passes
Remove trailing whitespace
Pythonify. Avoid ; and () around if statements

John (J5) Palmieri (1):
override that wasn’t checked in – fixes some test cases

Olav Vitters (1):
Fix doap file

Simon van der Linden (18):
Initial import
Add PyGObject patches
Add a doap file
Update PyGObject patches
Remove PyGObject patches since they’ve been merged to master
Fix silent rules setup
Import pygtk properly to avoid failure on some setups
Use the right variable when looking up in sys.modules
Fix members initialization in metaclasses
Remove global checks for pointers and move them in type cases
that need them
sys.path must be modified after pygtk is imported
Suppress compilation warnings
Don’t set a default constructor for structures.
Initialize struct fields to 0 when allocating
Restore the overrides support
Remove support for pointers to basic types as input-only
argument and return value
Fix and complete overrides tests
Add modelines and copyright information to overrides modules

Tomeu Vizoso (28):
Add myself to pygi.doap
Treat GI_INFO_TYPE_INTERFACE same as GI_INFO_TYPE_OBJECT
Register interfaces
Add support for Any arguments
Add stuff to .gitignore
Accept 0 as a valid value for flag and enum arguments
Structs in arrays are not marshalled correctly
A few tests about interfaces
Use the limit constants from glib and interpret G_MAXUINT32 as
PyLong_FromLongLong
Revert “Use the limit constants from glib and interpret
G_MAXUINT32 as PyLong_FromLongLong”
Set a default constructor for boxed structs that don’t have one
Use the limit constants from glib and interpret G_MAXUINT32 as
PyLong_FromLongLong
The array field ‘length’ starts to count from the C arg list, so
need to decrement when it’s a method
Add Gtk.keysyms to overrides
Always create the .so link
Use GIMarshallingTests (old TestGI) in gobject-introspection
Add metadata to the .doap file
Add gdb and valgrind variants for the tests
Allow creating structs with pointers
Add support for foreign structs
Add examples/cairo-demo.py
Require PyCairo
Update to latest version of the pygi-convert.sh script
Implement vfuncs.
Add missing file to tarballs
Add one more missing file to tarballs
Add more stuff to the tarballs
One more missing file…

Zach Goldberg (6):
Implementation callback support with scoping and basic argument support.
Add Zach Goldberg as a pygi maintainer
Move some tests from test_gi to test_everything
Implement nullable argument support, including tests
Fix a typo in pygi-callbacks.c header
Bump version for release 0.5.0

Gnome/Python Hackfest 2010 – Day 2, or: How to actually coordinate packages.

So it’s day two of the PyGI hackfest.  You can find some good coverage of Day 1 at Colin Walter’s Blog and John (J5) Palmieri’s Blog. The first few hours of day two include:

  • Made the decision that PyGI will depend on the GObject-Introspection 0.6.10 (to be released). This is because PyGI needs some new testing modules exported by GI as well as newer (faster!) APIs.
  • Maemo will originally get GI 0.6.7 and later be bumped to 0.6.10.
  • PyGI Callbacks patches all merged and are going through make check and patch review now
  • David Malcom got PyGI to compile and (mostly) link to Python 3.0
  • Tomeu got his hard drive to work
  • Colin Walters has been slacking
  • J5 has been compiling code all morning
  • We lack music. 🙁

Zhaan works with Sonos!

I finally brought my development N900 to the office to see if the Zhaan would play nice with Sonos. My initial efforts were thwarted and alas Sonos did not appear as either a source or a renderer. A quick glance at gupnp-universal-cp knocked me over the head with why — Sonos uses nested devices; something I had not previously tested Zhaan with.

A three line python hotfix later and Sonos now works wonderfully with Zhaan :). They curiously use REL_TIME instead of ABS_TIME for seeking and there is something funny with their implementation of GetVolume that Zhaan isn’t 100% happy with (SetVolume and Seek do work though) but on the whole definitely a success!

Update: A quick apt-get install mediatomb later (and a quick reconfig — mediatomb auto-bound to my VPN interface instead of my LAN, preventing anybody but my machine and the VPN machines, which my phone is not, from seeing it) and my desktop machine is now a perfectly good (and tested) sonos media source.  Nifty.  

Zhaan works on the N900!

It has been a long time coming but finally I’ve gotten Zhaan to work on the Nokia N900, a Maemo 5 device!  Today Anderson and Willer at PyMaemo handed me a whole bunch of armel package dependencies for Zhaan.  I combined them with some secret sauce (i.e. some extra projects, including updated GUPnP and GUPnPAV and their GIR/Typelib files), deployed it all to my development N900 and voila, we have a working Zhaan :).

If you wish to get this running yourself the only way right now is to do this manually.  There are still a whole bunch of packaging road blocks before this can make it to extras-devel.  Instructions to do it by hand:

WARNING: FOR ADVANCED USERS ONLY FOR NOW!  Following these instructions could potentially break some features of your device as it does override your built in gupnp, gupnp-av and pygobject libraries.  This will be made safe by the time it makes it to extras-devel.

Download http://www.zachgoldberg.com/data/ZhaanN900.tgz to your device and extract it.

Install all of the non-doc, non-dbg and non-dev “.deb files”.  e.g. “dpkg -i *.deb”.  (Note that the python-gobject-pygi may have a conflict and you may have to do dpkg -i –force-conflicts python-gobject-pygi…..).  Once all the debs are installed (I believe its 14 .deb files) copy the .gir files to /opt/pygi/share/gir-1.0/ and the .typelb files to /opt/pygi/lib/girepository-1.0/.

Finally download Zhaan from git (git://github.com/ZachGoldberg/Zhaan.git) and then use the following command to run zhaan:

LD_LIBRARY_PATH=/opt/pygi:/opt/pygi/lib PYTHONPATH=/opt/pygi/lib/python2.5/site-packages/gtk-2.0/ python2.5 zhaan.py

And finally, the obligatory screen shots.  (You can tell this is a real device because its charging and the battery guage is moving in the various screenshots, as well as the sim card icon).

Awesomeness.

GUPnP + PyGi Packaged for ARMEL — Maemo Extras here we come!

Just a quick update that I just heard word from the PyMaemo guys (Willer & Anderson Lizardo) that they managed to overcome the major obstacles in packaging the dependencies for Zhaan. I am going to work on these packages and make a new package for Zhaan and push everything to Maemo extras-devel within the next couple of days. Woohoo!

A *huge* thank you goes out to the PyMaemo folks for doing the packaging work on this. If you see one of those guys in the street please buy them a beer for me 🙂