arteme.fi

Kill That Hanging SSH!

arteme — Mon, 29 Mar 2010 15:12:44 +0000

EDIT: The scipt below was updated on 2010-04-08 to identify the
correct xterm-ssh pair based on the X window id.

I spend a lot of time working remotely, ssh-ing to hosts from places
with bad wireless connection or glitchy firewalls. When an ssh session
hangs, it is bad enough. Trying to locate one dead ssh amid a dozen
live ones is tedious. It grows into a major headache if you need to do
that several times a hour. So, I have a dead ssh in an xterm
window, how do I kill it? Moreover, how do I automate the process?

Luckily, most applications nowadays export the process id, controlling a
particular X window, as a property of that window and xprop is a tool
that can help us query that.

Run xprop as follows and click on the window of interest:

arteme@book:~$ xprop _NET_WM_PID
_NET_WM_PID(CARDINAL) = 17843

Knowing the process id of the terminal we can track down the ssh process
that runs within:

arteme@book:~$ pstree -Ap 17843
xterm(17843)---bash(17889)---ssh(21687)

There’s a process id that can be killed.

Half the work is done and it seems that it can be easily rolled into a shell
script, but if you did, you would notice that if you started an xterm within
an xterm and an ssh within that, pstree output would look like:

arteme@book:~$ pstree -Alp 3443
xterm(3443)---bash(3447)---xterm(3705)---bash(3706)---ssh(9612)

If you start the second xterm in background and start ssh in both of them,
the output becomes even more chaotic:

arteme@book:~$ pstree -Alp 3444
xterm(3444)---bash(3446)-+-ssh(12549)
                         `-xterm(12508)---bash(12509)---ssh(12540)

So, what to kill?

Luckily, you can trace the ssh process back to the terminal window it is
running in by it’s window id. Using xprop we can find out the window id
of the terminal, which is passed in the WINDOWID variable to the
processes started within. So, for the two xterms above with their
ssh:

arteme@book:~$ xprop -f WM_CLIENT_LEADER '32c' ' = $0\n' WM_CLIENT_LEADER
WM_CLIENT_LEADER(WINDOW) = 79691810
arteme@book:~$ cat /proc/12549/environ | tr '\0' '\n' | grep WINDOWID
WINDOWID=79691810

and:

arteme@book:~$ xprop -f WM_CLIENT_LEADER '32c' ' = $0\n' WM_CLIENT_LEADER
WM_CLIENT_LEADER(WINDOW) = 77594658
arteme@book:~$ cat /proc/12540/environ | tr '\0' '\n' | grep WINDOWID
WINDOWID=77594658

By default, “xprop” will return the window id as in hex, so the “-f …”
argument is necessary to override that and ease the comparison, since the
“WINDOWID” environment variable is in decimal.

Note, however, that if you run screen with multiple ssh instances within,
the environment of those ssh processes contains the WINDOWID of the
original xterm they were started in, if any. That information is not reliable
as it if you detached from the screen in one xterm and attached to it in
another one, the WINDOWID will no longer be correct. Luckily, one can
identify a screen session by the presence of the STY variable in the
environment…

Now this all can be rolled into a shell script:

#!/bin/bash

# Get the X terminal window properties and extract window id and process id
prop=`xprop -notype -f WM_CLIENT_LEADER '32c' ' = $0\n' \
            WM_CLIENT_LEADER \
            _NET_WM_PID`

wid=`echo $prop | sed -n 's/.*WM_CLIENT_LEADER = \([0-9]\{1,\}\).*/\1/p'`
pid=`echo $prop | sed -n 's/.*NET_WM_PID = \([0-9]\{1,\}\).*/\1/p'`

# Exit if any of the data was missing
[ -z "$wid" -o -z "$pid" ] && exit

# Get the process tree and extract the ssh process id from there
ssh=`pstree -Apl $pid | sed -n 's/.*ssh(\([0-9]\{1,\}\))/\1/p'`

# Filter ssh windows so that they do not belong to a screen session and
# do run in the X terminal the user clicked on
for i in $ssh; do
    sty=`cat /proc/$i/environ | tr '\0' '\n' | grep '^STY'`
    window=`cat /proc/$i/environ | tr '\0' '\n' |\
            sed -n 's/WINDOWID=\(.*\)/\1/p'`

    [ -n "$sty" ] && continue
    [ "$wid" != "$window" ] && continue

    tokill="$tokill $i"
done

# Exit if there's nothing to kill
[ -z "$tokill" ] && exit

# Kill the process
kill -9 $tokill

Grab that as a bash script, kill_that_ssh.sh. Furthermore, you can add it
to your quick-launch bar to never have to let go of your mouse when reaching
for the tool:

New Beginnings

arteme — Mon, 29 Mar 2010 14:46:24 +0000

In an attempt to clear out my head, I am back with things to post. Starting with a new iteration, you revise the tools and processes that have been holding you back. With this I have to say “good bye!” to trac that has been serving me faithfully as a platform for the site and say “hello!” to a shiny new Wordpress installation. This is ironic, since I moved to trac from Wordpress long time ago. Unfortunately, trac’s blogging and page infrastructure did not advance in the direction I would have liked to see it go, while Wordpress did. The latest addition that won me over was the ReST plug-in for Wordpress that I am happily using now- While I still frown upon running a MySQL server to power this tiny site, that is the way it is going to be for now.

I will be slowly migrating the content of the old site over to the new one and the old blog entries will eventually find their way to the new site too.

Django CherryPy Development Server and Serving Static Files

arteme — Thu, 26 Feb 2009 12:34:34 +0000

Introduction

Django web framework is famous for how easy it is to start developing
application with it. An installation of python and django is all you need to
get busy. Django provides you with all the essential gears and gadgets
including a simple development web server that you can use to run the
application you are developing, while you are developing it.

However, as your project grows more and more complex, you will discover the
limitations of the development server. The first one you are most likely to
face is the inability to serve static files (like site-specific media files).
To be frank, it is not so much an inability but reluctance — there is a
well-documented recepie of how to set up serving of static files, but it
has “do not use in production” written all over it. In fact, any use of the
development server in production environment is discouraged, as the following
disclaimer found in the documentation promptly states:

DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone through
security audits or performance tests. (And that’s how it’s gonna stay.
We’re in the business of making Web frameworks, not Web servers, so
improving this server to be able to handle a production environment is
outside the scope of Django.)

Another, more serious limitation is its inherent single-threadedness. If you
are developing any kind of AJAX application that might need to have more than
one connection to the web server at any point in time you will run into
problems — the development server will only start processing the second
request once it’s done with processing the first one. Recently, the topic of
a more advanced, multi-threaded development server has been coming up in the
mailing lists more and more often. All the pleads to improve the default dev.
server were met with “won’t fix” comments pointing out that it is extremely
easy to hookup a custom CherryPy server to the django WSGI request handler
achieving this functionality.

Basic CherryPy setup

As of version 1.0, Django provides an interface for adding custom management
commands, which can be used to integrate Django with CherryPy.
http://oebfare.com/blog/2008/nov/03/writing-custom-management-command/ is a
good article on how to integrate the two, but for the sake of clarity and
completeness let’s reiterate.

A word of warning: The code presented here is for CherryPy
version 3.1 since it uses WSGIPathInfoDispatcher class. It may be
possible to back-port it to version 3.0, but I make no guaranties.

We will add a “runcherrypy” command, so that the server is started as:

./manage.sh runcherrypy [-h ] [-p ]

The following file tree will be added to the project directory:

devserver/
 |-- __init__.py
 `-- management/
      |-- __init__.py
      `-- commands/
           |-- __init__.py
           `-- runcherrypy.py

where runcherrypy.py is as follows:

from django.core.management.base import BaseCommand
from django.core.handlers.wsgi import WSGIHandler

from cherrypy.wsgiserver import CherryPyWSGIServer

from optparse import OptionParser, make_option

class Command( BaseCommand ):

    option_list = BaseCommand.option_list + (
            make_option( "-h", "--host", dest="host", default="127.0.0.1" ),
            make_option( "-p", "--port", dest="port", default=8000 ),
            )

    def handle( self, *args, **options ):

        app = WSGIHandler()
        server = CherryPyWSGIServer( (options['host'], options['port']),
                                     app )

        try:
            server.start()
        except KeyboardInterrupt:
            server.stop()

    def create_parser( self, prog, subcommand ):
        """
        Create and return the ``OptionParser`` which will be used to
        parse the arguments to this command.
        """
        return OptionParser( prog=prog,
                             usage=self.usage(subcommand),
                             version=self.get_version(),
                             option_list=self.option_list,
                             conflict_handler="resolve" )

Now, add 'devserver' to your settings.py and you’re ready to go! If
you have set up your static file serving in django, as specified earlier,
everything should “just work”. But why go through the network of django
resolvers just to arrive at basic static page serving if you can do it on
the CherryPy side?

Serving static files

A careful observer might notice that django actually serves the admin
interface static content quite happily. That is done by
django.core.servers.basehttp.AdminMediaHandler class. Steven Wilcox
at http://www.devpicayune.com has already taken AdminMediaHandler into
use in his article on django/CherryPy integration. If only one could use
AdminMediaHandler to serve static content outside of the
.../contrib/admin/media/ directory…

Why not? Here is a modified version of AdminMediaHandler applicable for
serving any static content. We will put it to devserver/mediahandler.py:

import os, stat, mimetypes
import django
from django.utils.http import http_date
from django.conf import settings

class MediaHandler( object ):

    def __init__( self, media_root ):
        self.media_root = media_root

    def __call__( self, environ, start_response ):

        def done( status, headers, output ):
            start_response( status, headers.items() )
            return output

        path_info = environ['PATH_INFO']
        if path_info[0] == '/':
            path_info = path_info[1:]

        file_path = os.path.join( self.media_root, path_info )

        if not os.path.exists( file_path ):
            status = '404 NOT FOUND'
            headers = {'Content-type': 'text/plain'}
            output = ['Page not found: %%s' %% file_path]
            return done( status, headers, output )

        try:
            fp = open( file_path, 'rb' )
        except IOError:
            status = '401 UNAUTHORIZED'
            headers = {'Content-type': 'text/plain'}
            output = ['Permission denied: %%s' %% file_path]
            return done( status, headers, output )

        # This is a very simple implementation of conditional GET with
        # the Last-Modified header. It makes media files a bit speedier
        # because the files are only read off disk for the first request
        # (assuming the browser/client supports conditional GET).

        mtime = http_date( os.stat(file_path)[stat.ST_MTIME] )
        headers = {'Last-Modified': mtime}
        if environ.get('HTTP_IF_MODIFIED_SINCE', None) == mtime:
            status = '304 NOT MODIFIED'
            output = []
        else:
            status = '200 OK'
            mime_type = mimetypes.guess_type(file_path)[0]
            if mime_type:
                headers['Content-Type'] = mime_type
            output = [fp.read()]
            fp.close()

        return done( status, headers, output )

CherryPy version 3.1 provides WSGIPathInfoDispatcher class which exposes
an interface to run multiple WSGI applications side-by-side, distinguished
by request path. We will utilize this to set up three applications:

Django framework project, served by WSGIHandler: this will be placed
at webserver root, ‘/’.
Site media, served by MediaHandler: project settings file,
settings.py, defines MEDIA_ROOT and MEDIA_URL, which are the
path to the directory that hold the media and its URL on the server.
We will serve site media from that URL.
Admin media, served by MediaHandler: project settings file,
settings.py, defines ADMIN_MEDIA_PREFIX, which is the URL prefix
on the server. We will serve admin media from that URL.

This is how runcherrypy.py needs to be modified:

...
import django.contrib.admin

from cherrypy.wsgiserver import CherryPyWSGIServer, WSGIPathInfoDispatcher
from devserver.mediahandler import *
import os.path
...

    def handle( self, *args, **options ):

        app = WSGIHandler()

        path = { '/': app,
                 settings.MEDIA_URL: MediaHandler( settings.MEDIA_ROOT ),
                 settings.ADMIN_MEDIA_PREFIX:
                     MediaHandler(
                         os.path.join( django.contrib.admin.__path__[0],
                                       'media' )
                         )
               }
        dispatcher = WSGIPathInfoDispatcher( path )
        server = CherryPyWSGIServer( (options['host'], options['port']),
                                     dispatcher )

As you notice, we deduce the on-disk path of the admin media based on the
__path__ variable of the imported django.contrib.admin module.

Finally, a test application

First, a few practical thought on the layout of things. Django insists
that MEDIA_ROOT is an absolute path. If you are hosting your project
completely with CherryPy, you will probably keep all your media files
together with the project. I prefer to keep my media in a media/
directory within my project directory. settings.py features the
following code to resolve MEDIA_ROOT properly:

MEDIA_ROOT = 'media'

import os.path
MEDIA_ROOT = os.path.join(
               os.path.normpath( os.path.dirname(__file__) ),
               MEDIA_ROOT )

My applications start out from a simple structure:

alpha/
 |-- __init__.py
 |-- models.py
 |-- views.py
 |-- urls.py
 |-- media/
 |    `-- ...
 `-- templates/
      `-- ...

This way, I can create symbolic links in the project media/
directory like:

$ ls -l media/
total 0
lrwxrwxrwx 1 arteme users 13 Feb 27 09:34 alpha -> ../alpha/media/

which will make it easy to reference “alpha” application’s media in its
template files as “{{MEDIA_URL}}alpha/...“

Attached is a simple test application that put all these things together.
It features a project called cherrytest with a single application app.
The application has one view servers one HTML file with a 570×562 image
that is split into 35 100×100 tiles. Happy hacking!

cherrytest.tgz

Trac at site root with lighttpd 1.5.0

arteme — Mon, 20 Aug 2007 13:37:19 +0000

For a long time I have been trying to set up trac to work as site root.
Consider a website http://trac.site.com/, I would like to have URL
http://trac.site.com/ point to default page (usually the wiki page
also served as http://trac.site.com/wiki), URL
http://trac.site.com/timeline point to the timeline page,
URL http://trac.site.com/roadmap point to the roadmap page, etc.
At no point should trac rewrite the URLs to anything
like http://trac.site.com/trac/timeline.

Such setup is possible with an apache webserver, but has been extremely
difficult to get working with lighttpd. Fortunately, it is possible with
lighty version 1.5.0. Here is the configuration of a working setup featuring
lighttpd 1.5.0 Subversion snapshot of revision 1900 and trac 0.10.4.

Since it is lighttpd version 1.5.0, we use the new mod_proxy_core module
to communicate with the trac FastCGI process spawned externally using
the spawn-fcgi utility and listening for connections on localhost port
9000.

The layout of /var/www/trac.site.com/ directory is as follows:

/var/www/trac.site.com/
  |-- ...
  `-- htdocs
      |-- site-icon.ico
      |-- site-logo.png
      |-- ...
      `-- trac
      `-- ...

I want to serve static content from /var/www/trac.site.com/htdocs and
trac static content from /var/www/trac.site.com/htdocs/trac. Static
content includes website icon, website logo and whatever else I will manually
upload in the future. Trac static content is the files provided by a trac
installation and otherwise requested from .../chrome/common/.

This requires some modifications to the ini-file of the trac project.
My trac.ini has the following things set:

[header_logo]
src = /htdocs/site-logo.png
...

[project]
icon = /htdocs/site-icon.ico

[trac]
htdocs_location = /htdocs/trac/
...

Note: if you don’t set htdocs_location above, static trac content
will be requested from /chrome/common/ directory instead. In this
case, lighttpd will not handle the request itself, but will forward it
to trac. This will not break anything, as trac will serve it’s static
content without a glitch, but it may be slower in doing so than lighttpd.

The lighttpd configuration file looks like:

server.modules = ( "mod_alias",
                   "mod_auth",
                   "mod_proxy_code",
                   "mod_proxy_backend_fastcgi",
                   ...
                 )

...

$HTTP["host"] == "trac.site.com" {
  # trac-based website

  # static content root
  alias.url = ( "/htdocs/" => "/var/www/trac.site.com/htdocs/" )

  # authentication settings
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/var/trac/trac.htpasswd"

  $HTTP["url"] =~ "^/htdocs/" {
    # static concent
  }
  else $HTTP["url"] =~ "^/login$" {
    # login handling
    auth.require = ( "/" => ( "method"  => "basic",
                              "realm"   => "site.com trac website",
                              "require" => "valid-user"
                            )
                   )
  }

  $HTTP["url"] !~ "^/htdocs" {
    # trac via fastcgi
    proxy-core.protocol = "fastcgi"
    proxy-core.backends = ( "127.0.0.1:9000" )
    proxy-core.rewrite-request = ( "_pathinfo" => ( "^(/.*)" => "$1" ),
                                   "_scriptname" => ( "" => "/" )
                                 )
  }
}

...

The rewrite-request part doesn’t make sense, but it is the black magic
that makes it actually work.

I’m sure there are better ways of achieving the same behavior. If you
got the same result with lighttpd 1.5.0 with less hassle, I would like
to hear from you.

Maemo SDK on Gentoo/AMD64

arteme — Sun, 29 Apr 2007 13:20:05 +0000

Anyone who has tried installing Maemo SDK on a 64-bit system has run into
problems. Maemo SDK 2.2 and up (not that you would have more luck with
earlier versions) requires Scratchbox Apophis R4, which is 32-bit.

There are several pages on the web describing Maemo SDK installation on
Gentoo, 64-bit platform, or both:

This is my story of how I set up Maemo 3.1 SDK on Gentoo/AMD64…

First of all, you’ll need to download
http://repository.maemo.org/stable/3.1/maemo-scratchbox-install_3.1.sh and
http://repository.maemo.org/stable/3.1/maemo-sdk-install_3.1.sh

Step 1: Scratchbox

# uname -a

Linux harmony 2.6.20-suspend2-r5-ae070426 #1 PREEMPT Thu Apr 26 12:03:31
EEST 2007 x86_64 AMD Turion(tm) 64 Mobile Technology MT-37 AuthenticAMD
GNU/Linux

# linux32
# uname -a

Linux harmony 2.6.20-suspend2-r5-ae070426 #1 PREEMPT Thu Apr 26 12:03:31
EEST 2007 i686 AMD Turion(tm) 64 Mobile Technology MT-37 AuthenticAMD
GNU/Linux

# bash ~/maemo-scratchbox-install_3.1.sh -s /maemo-3.1/scratchbox

...

# /maemo-3.1/scratchbox/sbin/sbox_adduser arteme yes

Scratchbox user account for user arteme added

Step 2: Maemo SDK

After you have added yourself as a user to sbox group, you will need to
start a new terminal, re-login, or whatever else you do to get the group
membership updated. I simply do:

# ssh 127.0.0.1

Then:

# bash maemo-sdk-install_3.1.sh -s /maemo-3.1/scratchbox

Step 3: Xephyr

Xephyr is available in Gentoo as a kdrive USE-flag for xorg-server.
Although if you want to run it natively in 64-bit mode, you’re out of luck!
At the time of writing, there are a number of bugs that make X crash at
startup or otherwise. In particular,
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/66996
(or https://bugs.freedesktop.org/show_bug.cgi?id=8417 in freedesktop.org
bugzilla) and others. Certain distributions have patched the bugs themselves
(http://cvs.fedora.redhat.com/viewcvs/rpms/xorg-x11-server/devel/) and there
are some fixes upstream
(http://gitweb.freedesktop.org/?p=xorg%%2Fxserver.git&a=search&s=ephyr).
However, for a painless installation I have used a 32-bit chroot environment.

32-bit chroot environment

You can follow the 32-bit Chroot Guide for Gentoo/AMD64 or use the
x86-chroot package:

# emerge x86-chroot
# rc-update add x86-chroot default
# /etc/init.d/x6-chroot start

Xephyr in 32-bit chroot environment

Enter the 32-chroot environment:

# x86-chroot

Install Xephyr:

# USE="kdrive -ipv6 -dri" VIDEO_CARDS="vga" emerge xorg-server

You’re all set!

Everything is installed, you’re all set! Don’t forget to start scratchbox on
system start-up:

# /maemo-3.1/scratchbox/sbin/sbox_ctl start

Start Xephyr. You can do that through the x86-chroot as root, or
better as a user:

$ LD_LIBRARY_PATH=/opt/x86-chroot/lib:/opt/x86-chroot/usr/lib \
  /opt/x86-chroot/usr/bin/Xephyr :2 -host-cursor \
                                    -screen 800x480x16 -dpi 96 -ac

Log into scratchbox:

$ /maemo-3.1/scratchbox/login

Welcome to Scratchbox, the cross-compilation toolkit!

Use 'sb-menu' to change your compilation target.
See /scratchbox/doc/ for documentation.

[sbox-SDK_X86: ~] > export DISPLAY=:2
[sbox-SDK_X86: ~] > af-sb-init.sh start
...

Subversion-only SSH users

arteme — Thu, 26 Apr 2007 13:50:47 +0000

Sometimes you’re stuck with svn+ssh://… access for your repository. You need to give someone SSH access to access the repository, but you don’t want to give them full shell access. What you can do is create a group, whose users will only have access to svnserve.

Create a group called “svnonly”. /etc/groups will look something like:

...
svnonly:x:414
...

Then you can add users into that group. /etc/passwd will look something like:

...
john:x:1001:414::/home/svnonly/john:/bin/bash
mike:x:1002:414::/home/svnonly/mike:/bin/bash
bob:x:1003:414::/home/svnonly/bob:/bin/bash
...

The home directory is entirely up to you – I choose /home/svnonly as a top directory for svnonly-users to keep them separate from full-access users and to keep /home neat. Yet, each can have their own home directory to have .ssh/authorized_keys, for example.

Next, add the following to your /etc/ssh/sshd_config:

Match Group svnonly
        X11Forwarding no
        AllowTcpForwarding no
        ForceCommand svnserve -t

Now svnonly-users will have svnserve started for them automatically when they connect instead of a normal shell.

Rendering an SVG to a GTK image

arteme — Sun, 25 Mar 2007 22:00:49 +0000

Once upon a time loading SVG in python was done via pycairo and libsvg-cairo:

import cairo
import cairo.svg

svg = cairo.svg.Context()
svg.parse( 'filename.svg' )
...

Since then, libsvg-cairo has been deprecated (http://cairographics.org/PycairoFAQ). Nowadays, librsvg provides an SVG loader for GTK, so that it is possible to simply do

import gtk

i = gtk.Image()
i.set_from_file( 'x.svg' )
...

But I needed a finer control over the generation of image – in particular, I wanted to be able to zoom in to and out of the vector graphics image, much like rsvg-view. If you follow the example above and first render the image, then start scaling it, you will not get a high-resolution scaled vector image, but a scaled low-resolution rendering of the vector image.

After reading the rsvg-view sources, I figured out that what I wanted could be accomplished by:

Request an SVG loader from GTK/GDK
Set the desired image size
Load the image
Set the image to the resulting pixbuf

The following code does the magic:

# svg_data - a string containing the contents of the SVG file
# image - an instance of gtk.Image
# w, h - desired image resolution

loader = gtk.gdk.PixbufLoader( 'svg' )
loader.set_size( w, h )

loader.write( svg_data )
loader.close()

pb = loader.get_pixbuf()
image.set_from_pixbuf( pb )