Note: All my config dotfiles are now in a repository at GitHub.

Mostly the same as last time, with a few updates from the last few years of working in the real world. Important additions are the ignorecase/smartcase/incsearch to improve searching, adding a new “rr” code that inserts the regex to remove trailing spaces, and the final line will set vim to use the CTAGS files specified in the shell envvar $CTAGS_DB. The four let lines in the middle are supposed to change your terminal cursor based on your mode, but sometimes it seems to mess up the terminal so your mileage may vary. Includes auto indent and auto syntax highlighting. I use a black background terminal, so the “set bg=dark” optimizes the colors for a dark background. I like 4 spaces for my indenting, no tabs. I use the F5 key to toggle search result highlighting, and F6 to toggle between paste mode (no auto-indent) and no-paste mode.

set backupcopy=yes
set softtabstop=4
set autoindent
set cindent
set shiftwidth=4
set shiftround
set expandtab
set formatoptions=t
set textwidth=0
set mouse=a
set backspace=indent,eol,start
set ignorecase
set smartcase
set incsearch

map <f5> :set hls!<bar>set hls?<CR>
map <f6> :set paste!<bar>set paste?<CR>

let &t_ti.="\e[1 q"
let &t_SI.="\e[5 q"
let &t_EI.="\e[1 q"
let &t_te.="\e[0 q"

syntax enable
set background=dark

nmap rr :%s/\s\+$//g

let &tags=$CTAGS_DB

My mom used to make this Pozole when I was a kid, and I really enjoyed it. Good strong flavors with fun accessories at serving time. Good use of tomatillos.

Pozole

  • 2 T. cooking oil
  • 2 pound pork roast – Trimmed and cut into 1 1/2″ cubes
  • 1 large onion, minced
  • 4 garlic cloves, minced
  • 5 Cups chicken stock
  • 10 tomatillos, husked, cored, and quartered
  • 1 bunch fresh cilantro, stemmed (de-stemmed?)
  • 1 15oz. can hominy, drained and rinsed
  • 2 dried red chilies
  • Salt, pepper
  • Garnish:
    • 1 med onion finely diced
    • Shredded lettuce
    • Thinly sliced radishes
    • Lime wedges

Heat oil in large heavy skillet (Dutch oven) over high heat. Add pork and cook until brown on all sides.

Reduce heat to medium. Add large onion and cook until tender stirring occasionally, about 10 minutes.

Add garlic and cook 2 min.

Add 3 cups stock. Simmer until meat is very tender, about 90 minutes.

Puree tomatillos and cilantro with remaining 2 cups stock in blender. Add puree, hominy, and red chilies to pork. Simmer 15 min. Season with salt and pepper.

(Can be prepared 2 days ahead. Cover and refrigerate. Rewarm over low heat)

Discard Chilies. Serve pozole, passing diced onion, lettuce, radishes and line separately.

A friend was looking for a way to list the space usage on a windows server that only had FTP access. I had written something similar for a project long ago, and polished up to do the job.

This python script will walk an FTP directory in a top-down, depth-first pattern. It uses the ftplib library, which I believe is built-in to most or all python distributions. Configure the FTP_* variables near the top to set the server, port, user, password, and the delay between each FTP operation (to avoid hammering the server). The script recursively processes directories, creating a dirStruct tuple that contains the following items:

(pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)
    pwd is a string like "/debian/dists/experimental"
    subdirList is a list of tuples just like this one
    fileList is a list of (filename, sizeInBytes) tuples
    sizeInFilesHere is a sum of all the files in this directory
    sizeTotal is a sum of all the files in this directory and all subdirectories

It also writes data to two CSV files:

  • dirStruct_only_folders.csv
    • Contains entries for just the directories.
    • Local size is the total size of files in that folder (does not count subdirs).
    • Total size is the sum of local size and total size of all subdirs.
  • dirStruct_complete.csv
    • Contains entries for both files and folders.
    • Files do not have a total size, only a local size.
#!/usr/bin/env python
#
# A script to recursively walk an FTP server directory structure, recording information
# about the file and directory sizes as it traverses the folders.
#
# Stores output in two CSV files:
#  dirStruct_only_folders.csv
#     Contains entries for just the directories.
#     Local size is the total size of files in that folder (does not count subdirs).
#     Total size is the sum of local size and total size of all subdirs.
#  dirStruct_complete.csv
#     Contains entries for both files and folders.
#     Files do not have a total size, only a local size.
#
# Customize the FTP_* variables below.
#
# Basically does a depth-first search.
#
# Written by Matthew L Beckler, matthew at mbeckler dot org.
# Released into the public domain, do whatever you like with this.
# Email me if you like the script or have suggestions to improve it.

from ftplib import FTP
from time import sleep


FTP_SERVER = "ftp.debian.org"
FTP_PORT = "21" # 21 is the default
FTP_USER = "" # leave empty for anon FTP server
FTP_PASS = ""
FTP_DELAY = 1 # how long to wait between calls to the ftp server

def parseListLine(line):
   # Files look like          "-rw-r--r--    1 1176     1176       176158 Mar 30 01:52 README.mirrors.html"
   # Directories look like    "drwxr-sr-x   15 1176     1176         4096 Feb 15 09:22 dists"
   # Returns (name, isDir, sizeBytes)
   items = line.split()
   return (items[8], items[0][0] == "d", int(items[4]))

# Since the silly ftp library makes us use a callback to handle each line of text from the server,
# we have a global lines buffer. Clear the buffer variable before doing each call.
lines = []
def appendLine(line):
   global lines
   lines.append(line)
def getListingParsed(ftp):
   """ This is a sensible interface to the silly line getting system. Returns a copy of the directory listing, parsed. """
   global lines
   lines = []
   ftp.dir(appendLine)
   myLines = lines[:]
   parsedLines = map(parseListLine, myLines)
   return parsedLines
   
def descendDirectories(ftp):
   # Will return a tuple for the current ftp directory, like this:
   # (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)
   #     pwd is a string like "/debian/dists/experimental"
   #     subdirList is a list of tuples just like this one
   #     fileList is a list of (filename, sizeInBytes) tuples
   #     sizeInFilesHere is a sum of all the files in this directory
   #     sizeTotal is a sum of all the files in this directory and all subdirectories

   sleep(FTP_DELAY) # be a nice client

   # make our directory structure to return
   pwd = ftp.pwd()
   subdirList = []
   fileList = []
   sizeInFilesHere = 0
   sizeTotal = 0

   print pwd + "/"
   items = getListingParsed(ftp)
   for name, isDir, sizeBytes in items:
      if not isDir:
         fileList.append( (name, sizeBytes) )
         sizeInFilesHere += sizeBytes
      else:
         # is a directory, so recurse
         ftp.cwd(name)
         struct = descendDirectories(ftp)
         ftp.cwd("..")
         subdirList.append(struct)
         sizeTotal += struct[4]

   # add in the size of all files here to sizeTotal
   sizeTotal += sizeInFilesHere
   return (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)

def pprintBytes(b):
   """ Pretty prints a number of bytes with a proper suffix, like K, M, G, T. """
   suffixes = ["", "K", "M", "G", "T", "?"]
   ix = 0
   while (b > 1024):
      b /= 1024.0
      ix += 1
   s = suffixes[min(len(suffixes) - 1, ix)]
   if int(b) == b:
      return "%d%s" % (b, s)
   else:
      return "%.1f%s" % (b, s)

def pprintDirStruct(dirStruct):
   """ Pretty print the directory structure. RECURSIVE FUNCTION! """
   print "{}/ ({} in {} files here, {} total)".format(dirStruct[0], pprintBytes(dirStruct[3]), len(dirStruct[2]), pprintBytes(dirStruct[4]))
   for ds in dirStruct[1]:
      pprintDirStruct(ds)

def saveDirStructToCSV(dirStruct, fid, includeFiles):
   """ Save the directory structure to a CSV file. RECURSIVE FUNCTION! """
   # Info about this directory itself
   fid.write("\"{}/\",{},{}\n".format(dirStruct[0], dirStruct[3], dirStruct[4]))
   pwd = dirStruct[0]

   # Info about files here
   if includeFiles:
      for name, size in dirStruct[2]:
         fid.write("\"{}\",{},\n".format(pwd + "/" + name, size))

   # Info about dirs here, recurse
   for ds in dirStruct[1]:
      saveDirStructToCSV(ds, fid, includeFiles)

print "Connecting to FTP server '%s' port %s..." % (FTP_SERVER, FTP_PORT)
ftp = FTP()
ftp.connect(FTP_SERVER, FTP_PORT)
if FTP_USER == "":
   ftp.login()
else:
   ftp.login(FTP_USER, FTP_PASS)

print "Walking directory structure..."
dirStruct = descendDirectories(ftp)

print ""
print "Finished descending directories, here is the info:"
pprintDirStruct(dirStruct)
print ""

FILENAME = "dirStruct_complete.csv"
print "Saving complete directory info (files and folders) to a CSV file: '%s'" % FILENAME
with open(FILENAME, "w") as fid:
   fid.write("\"Path\",\"Local size\",\"Total size\"\n")
   saveDirStructToCSV(dirStruct, fid, includeFiles=True)

FILENAME = "dirStruct_only_folders.csv"
print "Saving directory info (only folders) to a CSV file: '%s'" % FILENAME
with open(FILENAME, "w") as fid:
   fid.write("\"Path\",\"Local size\",\"Total size\"\n")
   saveDirStructToCSV(dirStruct, fid, includeFiles=False)

Sample CSV output:

"Path","Local size","Total size"
"/plugins/",5426535,7594527
"/plugins/foo-1.1.jar",7774,
"/plugins/CHANGELOG.txt",45169,

Local size is just the size of the file itself, or the size of all files in a directory. Total size is the total size of the files in a directory plus the total sizes of all subdirectories. Files do not have a total size entry.

I recently discovered that two of the hard drives in my server had a firmware bug that could leave to silent dataloss when used with smartd (and other programs). Fortunately, there is a firmware update. Unfortunately, it doesn’t change the drive’s reported firmware revision so you can’t tell if the update has been applied already…

To run the update, it says “Save the .exe files to a bootable media”. It doesn’t provide any more details than that, but apparently it needs to be a DOS boot disk/usb drive. This link provides an easy way to take a pre-generated FreeDos image and DD it to a USB drive. Once you do that, you can mount it and copy the files to run to the FAT partition.

FreeDOS prebuilt bootable USB flash drive image – http://chtaube.eu/computers/freedos/bootable-usb/

As part of migrating to a new account at Dreamhost, I had to regenerate my blog from an old SQL dump backup email. Not quite as easy as from an actual WordPress export, but I made it work. However, something got messed up with the RSS feed. Turns out my old blog install had the following in the /blog/.htaccess that didn’t make it into the new install:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /blog/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]
</IfModule>

Adding it to the same place in the new install fixed things, but changed the location to http://www.mbeckler.org/blog/?feed=rss2 . Just an FYI.

I’m writing a fun little webapp using Flask and Python and Sqlalchemy, running on Heroku using a PostgreSQL database. I use a sqlite3 database file for local testing, and PostgreSQL when I deploy, so naturally there are some minor snags to be run into when switching between database engines.

Tonight I ran into a tricky issue after adding a ton more foreign-key / relationships to my database-backed models. I was getting an error like this when I tried to issue my db.drop_all() command in my python script that initializes my database tables:

sqlalchemy.exc.InternalError: (InternalError) cannot drop table pages because other objects depend on it
DETAIL:  constraint pagesections_parent_page_id_fkey on table pagesections depends on table pages
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
 '\nDROP TABLE pages' {}

A bunch of searching for solutions indicated that maybe it would work if you run db.reflect() immediately before the db.drop_all(), but apparently the reflect function is broken for the current flask/sqlalchemy combination. Further searching revealed a mystical “DropEverything” function, and I finally found a copy here. I had to do a few small modifications to get it to work in the context of Flask’s use of Sqlalchemy.

def db_DropEverything(db):
    # From http://www.sqlalchemy.org/trac/wiki/UsageRecipes/DropEverything

    conn=db.engine.connect()

    # the transaction only applies if the DB supports
    # transactional DDL, i.e. Postgresql, MS SQL Server
    trans = conn.begin()

    inspector = reflection.Inspector.from_engine(db.engine)

    # gather all data first before dropping anything.
    # some DBs lock after things have been dropped in 
    # a transaction.
    metadata = MetaData()

    tbs = []
    all_fks = []

    for table_name in inspector.get_table_names():
        fks = []
        for fk in inspector.get_foreign_keys(table_name):
            if not fk['name']:
                continue
            fks.append(
                ForeignKeyConstraint((),(),name=fk['name'])
                )
        t = Table(table_name,metadata,*fks)
        tbs.append(t)
        all_fks.extend(fks)

    for fkc in all_fks:
        conn.execute(DropConstraint(fkc))

    for table in tbs:
        conn.execute(DropTable(table))

    trans.commit()

I had to change the uses of engine to db.engine since Flask’s SQLalchemy takes care of that for you. You get the db object from the app, like this “from myapp import db”, and this is how I defined db in myapp:

from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__, etc)

# DATABASE_URL is set if we are running on Heroku
if 'DATABASE_URL' in os.environ:
    app.config['HEROKU'] = True
    app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL']
else:
    app.config['HEROKU'] = False
    app.config['SQLALCHEMY_DATABASE_URI'] = "sqlite:///" + os.path.join(PROJECT_ROOT, "../app.db")

db = SQLAlchemy(app)

And then this is the important parts of my db_create.py script:

from sqlalchemy.engine import reflection
from sqlalchemy.schema import (
        MetaData,
        Table,
        DropTable,
        ForeignKeyConstraint,
        DropConstraint,
        )

from cyosa import app, db

if not app.config['HEROKU'] and os.path.exists("app.db"):
    os.remove("app.db")

def db_DropEverything(db):
    # listed above

db_DropEverything(db)
db.create_all()

# add your instances of models here, be sure to db.session.commit()

I wanted to play Minecraft on my 64-bit Ubuntu Linux install, but it wasn’t working correctly for me, and would give a black screen after login, and the console reported some errors about xrandr (which might be related to my odd “dual display-port + docking station” setup at home). After some searching, I found a tip to manually install the LWJGL java libraries into the ~/.minecraft/bin/ folder, to have the latest and greatest version of those libraries.

  1. Download the latest version zip archive of the LWJGL libraries: http://sourceforge.net/projects/java-game-lib/files/latest/download?source=files
  2. Extract downloaded zip archive
  3. Copy all files in lwjgl-2.9/jar/ to ~/.minecraft/bin/
  4. Copy all files in lwjgl-2.9/native/linux/ to ~/.minecraft/bin/natives/

And then you should be good to go.

via https://bbs.archlinux.org/viewtopic.php?pid=876274#p876274

Every stinkin’ time you have to update Java (which is every day it seems, since Java has more (security) holes than most Swiss cheese) it wants to install the stupid Ask Toolbar and take over your default search engine. Here’s a quick Windows registry fix that will apparently disable the installer from even asking you about the toolbar installation.

Another way, without having to download and rename or create a new .REG file, is to copy and paste the following two lines into an elevated CMD prompt:

reg add HKLM\software\javasoft /v "SPONSORS" /t REG_SZ /d "DISABLE" /f

reg add HKLM\SOFTWARE\Wow6432Node\JavaSoft /v "SPONSORS" /t REG_SZ /d "DISABLE" /f

via Superuser: How can I prevent Ask.com Toolbar from being installed every time Java is updated?

Update: Looks like Transmission sends traffic out the loopback (lo) interface, to the loopback interface. Seems kind of weird, but it should be harmless. These rules should permit traffic from the vpnroute gid to pass to the tun0 and lo interfaces, but everything else will be rejected. You can also duplicate the last rule with a LOG target if you want to see what is still being rejected.

sudo iptables -A OUTPUT -m owner --gid-owner vpnroute -o tun0 -j ACCEPT

sudo iptables -A OUTPUT -m owner --gid-owner vpnroute -o lo -j ACCEPT

sudo iptables -A OUTPUT -m owner --gid-owner vpnroute -j REJECT


We recently moved into a new home where we have a shared internet connection with the other occupants of the duplex. I didn’t want to use bittorrent directly, since any nastygrams would end up with the landlord and cause problems, so I signed up for the IPredator VPN service in Sweden. It allows you to make an encrypted and secure connection from your computer to their network, so all of your internet traffic is funneled through the secure connection, making it so that the neighbors, landlord, and internet service provider can’t tell what I’m up to. The VPN was really easy to set up in Ubuntu Linux with the graphical network manager (IPredator provides a visual guide to this process) and the speeds are certainly reasonable.

One downside of this is that if there is a connection hiccup that causes the VPN to drop, the bittorrent software will just fall back to sending data out the regular, unencrypted network interface, potentially exposing your naughty activities to the ISP. I wanted to find a way to effectively say, “only allow bittorrent traffic through the VPN connection” that would step up and protect things if the VPN connection dropped.

On Linux, the standard firewall is called “iptables”, and can do just what we need, in only three commands. But first, a couple of assumptions:

  • I am assuming that you are using the default Ubuntu Linux bittorrent client called “Transmission”, which is executed using the command “transmission-gtk”.
  • When the VPN is connected, it creates a new network interface called “tun0” (“tun” for “tunnel”).

The general plan is to somehow tag the bittorrent traffic so that the iptables firewall can identify the relevant packets, and reject them if they aren’t heading out the secure VPN interface tun0. An easy way is to run your bittorrent program using a different UNIX user or group.

Here, we add a new group called “vpnroute”:

sudo groupadd vpnroute

Then, we add the firewall rule that rejects all traffic from this group that is not heading out tun0:

sudo iptables -A OUTPUT -m owner --gid-owner vpnroute \! -o tun0 -j REJECT

Finally, we start up the bittorrent software with group ownership of vpnroute:

sudo -g vpnroute transmission-gtk

Your torrents should now only run when the VPN is connected. Try it out with some safe torrents, like the Ubuntu ISO files, and make sure that they only download when the VPN is connected, and they should stop right away when you disable the VPN.


If you want to confirm that the firewall rule is actually matching your traffic, you can add a similar rule that does the LOG operation instead of REJECT. You need to ensure that the LOG rule happens first, because after handling the LOG rule the packet keeps going, while a REJECT action won’t let the packet continue down the chain of rules. You can remove the output rule with “sudo iptables -F OUTPUT” (F for Flush), and then:

sudo iptables -A OUTPUT -m owner --gid-owner vpnroute \! -o tun0 -j LOG
sudo iptables -A OUTPUT -m owner --gid-owner vpnroute \! -o tun0 -j REJECT

Then you can check the output of “dmesg” to see when packets are logged (and then rejected) by the firewall.

Being from the Minnesota/Wisconsin area, but living in Pittsburgh, we tend to drive through Chicago 2-3 times each year. The GPS and online mapping services suggest taking I-90 all the way through Chicago, taking the Chicago Skyway to the Dan Ryan to the Chicago Circle to the Kennedy Expressway to Rockford. While that route is indeed the shortest highway route in terms of distance, it’s almost guaranteed to be very congested with lots of traffic, and ends up being slower in the end compared to other routes.

Over the past five years of driving through Chicago a few times per year, we’ve decided that taking I-290 and I-294 is the best way to bypass as much Chicago traffic as possible, without increasing the overall driving distance too much.

chicago_route_overview

Westbound Route Details – Heading from Indiana to Wisconsin, first you need to exit from the Indiana Toll Road (I-80W / I-90W) onto I-80W / I-94W, which is kind of a goofy right-exit-then-overpass-to-the-left sort of thing:

chicago_route_westbound_indiana

I-94W will peel off after a while and head North into Chicago. The transition from I-80W to I-294N is really easy, as I-80W has to exit and cross over to head further West, while you just stay in the center lanes for the curve toward the North.

The transition from I-294N to I-290W is somewhat tricky, and is usually the only place we encounter any traffic-based slowdown. It’s a silly right-exit loop-under, that sometimes gets backed-up a few hundred feet owing to the slow speeds on the tight loop. Merging with the I-290W traffic isn’t too bad.

chicago_route_westbound_294_to_290

After that, the transition from I-290W to I-90W is a standard cloverleaf loop, and it has some nice protected “feeder” lanes to make your merge really easy.

chicago_route_westbound_290_to_90

Eastbound Route Details – Heading from Wisconsin to Indiana, first you need to exit from I-90E to I-290E, which is a simple right exit, so there’s no picture here.

The transition from I-290E to I-294S is much simpler than the opposite-direction transition loop-around shown above, and is just a simple “keep right” sort of bump around the interchange.

chicago_route_eastbound_290_to_294

The transition from I-294S to I-80E is trivial, as is the interchange where I-94E merges into I-80E. The only tricky bit remaining is near the Indiana state line where you merge from I-80E / I-94E onto I-80E / I-90E, which has a silly cloverleaf bridge thing.

chicago_route_eastbound_indiana

Overall, a pretty easy route, usually with no significant traffic that would be easy to avoid.

There are definitely some tolls along the way, but I think all the Illinois toll plazas have open-road tolling, so get your EZ-Pass and save time and money. If you get your EZ-Pass from Pennsylvania (you don’t have to be a PA resident) and elect for paperless billing and connect your credit card for automatic reloading of funds, there are no monthly or yearly fees!