Introducing the Personal *nix Cluster Manager

I was at home this holiday, with the family enjoying the Thanksgiving Holiday.  I was messing around with stuff on my netbook and realized I wanted files from my desktop back in my apartment.   Only problem was — I have no idea what my IP is back there (I moved recently).  I’m also too lazy to setup my own or pay for a proper dyndns solution.  Why is it that in 2010 I still have this kind’ve problem?  Shouldn’t there be some application I can run on all my client pcs which keep them meshed together?

A bit of digging, couldn’t find anything.  So I wrote one.  Introducing the “Personal Cluster Manager for *nix/ssh systems”.

First, an important assumption: This software is meant to be used by you and you only.  It probably won’t scale to 1000 machines and I’m perfectly fine with that.  I have a handful (10?) machines I care about keeping track of for my stuff and I imagine most of the people who are interested in this are on the same order of magnitude.

Ok, enough nonsense, what is it?  There are three pieces:

  1. A client daemon that maintains a reverse ssh tunnel with a master, always connected and accessible node.  Theres a good bit of configuration to get this right, so theres also a little helper script to ask the user questions to get the right config values for the client.
  2. A “master” script.  This provides  an interface to a sqlite database which keeps track of what hosts we’ve seen, what ports their tunnels are on etc.  You can ask the master “show me all hosts you know about” or “which tunnels are active and usable right now?”.
  3. A curses based UI.  The UI pulls all its data from the master program, and doesn’t do any direct querying.  (This is intentional, as I intend to build other kinds of UIs and want all the data stuff to be handled underneath the UI).  The UI shows you what tunnels are active to which hosts, and with one keypress can give you a shell on any of the remote machines (using the reverse tunnel setup by the client).

All 3 of the above are working in version 0.1 in github.

A very poor quality screencast with typos and minor bugs left and right, but you’ll get the idea: (nevermind the ‘scrollz’ stuff, thats something particular about my local bashrc and nothing to do with PCM)

7 comments

  1. Hi, nice work, I’m working on a project to solve my desktop/laptop/where-are-my-files-again problem. have a look at https://github.com/thouters/ltr

  2. OpenVPN is your friend ;)

  3. Not too bad looking, but it seems as though you’ve written a less scalable version of mcollective.

    • Kind’ve ish sorta… my goal here isn’t to really run commands in parallel, but to give the admin the ability to access in anyway he/she sees fit an arbitrary one machine at a time.

  4. FWIW, I’m using dyndns and I don’t pay anything for it. Since *years*.

  5. ssh tunnels can tend to be a little slow, so take a look and Anders’ suggestion of using OpenVPN for the network part, or also N2N which is UDP-based and tends to have lower latency and allow clients to move to different physical networks transparently.

    E.g. have the N2N running on my own and my wife’s computer and can ssh into/from both of these no matter where the machines are, as long as they have access to the supernode.

    Similarly I have a samba filesystem mounted on my laptop, served from an office machine on an 10.2.1.X IP address and have an editor have files open on the mounted filesystem. Using ssh before needed a client tunnel restart before.

    /enough rambling :-)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>