Archives

All posts by Matthew

This is one of my favorite passages from Cryptonomicon, which I think about every time I see a machine struggling with its task.

Now when Bobby Shaftoe had gone through high school, he’d been slotted into a vocational track and ended up taking a lot of shop classes. A certain amount of his time was therefore, naturally, devoted to sawing large pieces of wood or metal into smaller pieces. Numerous saws were available in the shop for that purpose, some better than others. A sawing job that would be just ridiculously hard and lengthy using a hand saw would be accomplished with a power saw. Likewise, certain cuts and materials would cause the smaller power saws to overheat or seize up altogether and therefore called for larger power saws. But even with the biggest power saw in the shop, Bobby Shaftoe always got the sense that he was imposing some kind of stress on the machine. It would slow down when the blade contacted the material, it would vibrate, it would heat up, and if you pushed the material through too fast it would threaten to jam. But then one summer he worked in a mill where they had a bandsaw. The bandsaw, its supply of blades, its spare parts, maintenance supplies, special tools and manuals occupied a whole room. It was the only tool he had ever seen with infrastructure. It was the size of a car. The two wheels that drove the blade were giant eight-spoked things that looked to have been salvaged from steam locomotives. Its blades had to be manufactured from long rolls of blade-stuff by unreeling about half a mile of toothed ribbon, cutting it off, and carefully welding the cut ends together into a loop. When you hit the power switch, nothing would happen for a little while except that a subsonic vibration would slowly rise up out of the earth, as if a freight train were approaching from far away, and finally the blade would begin to move, building speed slowly but inexorably until the teeth disappeared and it became a bolt of pure hellish energy stretched taut between the table and the machinery above it. Anecdotes about accidents involving the bandsaw were told in hushed voices and not usually commingled with other industrial-accident anecdotes. Anyway, the most noteworthy thing about the bandsaw was that you could cut anything with it and not only did it do the job quickly and coolly but it didn’t seem to notice that it was doing anything. It wasn’t even aware that a human being was sliding a great big chunk of stuff through it. It never slowed down. Never heated up.

In Shaftoe’s post-high-school experience he had found that guns had much in common with saws. Guns could fire bullets all right, but they kicked back and heated up, got dirty, and jammed eventually. They could fire bullets in other words, but it was a big deal for them, it placed a certain amount of stress on them, and they could not take that stress forever. But the Vickers in the back of this truck was to other guns as the bandsaw was to other saws. The Vickers was water-cooled. It actually had a fucking radiator on it. It had infrastructure, just like the bandsaw, and a whole crew of technicians to fuss over it. But once the damn thing was up and running, it could fire continuously for days as long as people kept scurrying up to it with more belts of ammunition. After Private Mikulski opened fire with the Vickers, some of the other Detachment 2702 men, eager to pitch in and do their bit, took potshots at those Germans with their rifles, but doing so made them feel so small and pathetic that they soon gave up and just took cover in the ditch and lit up cigarettes and watched the slow progress of the Vickers’ bullet-stream across the roadblock. Mikulski hosed down all of the German vehicles for a while, yawing the Vickers back and forth like a man playing a fire extinguisher against the base of a fire. Then he picked out a few bits of the roadblock that he suspected people might be standing behind and concentrated on them for a while, boring tunnels through the wreckage of the vehicles until he could see what was on the other side, sawing through their frames and breaking them in half. He cut down half a dozen or so roadside trees behind which he suspected Germans were hiding, and then mowed about half an acre of grass.

crappy_cardboard_box_set
For Christmas 2015 I wanted to get the Harry Potter Hardcover Box Set for my dearest wife. The official box set hardcovers are very expensive ($120+) and come in a pretty cheap-looking cardboard “trunk” (photo at right). Plus, we already had all the books in hardcover, so I thought I would make a cute little wooden trunk myself. Everyone loves homemade gifts.

Lots of time and money later, I had finished a nice trunk, perfectly-sized for all seven HP hardcovers. Like all projects, I should have taken more photos during the process, but hopefully this write-up gives you a decent idea of what I did and how you could do the same if you want.

books_in_trunk

Continue Reading

Note: All my config dotfiles are now in a repository at GitHub.

Mostly the same as last time, with a few updates from the last few years of working in the real world. Important additions are the ignorecase/smartcase/incsearch to improve searching, adding a new “rr” code that inserts the regex to remove trailing spaces, and the final line will set vim to use the CTAGS files specified in the shell envvar $CTAGS_DB. The four let lines in the middle are supposed to change your terminal cursor based on your mode, but sometimes it seems to mess up the terminal so your mileage may vary. Includes auto indent and auto syntax highlighting. I use a black background terminal, so the “set bg=dark” optimizes the colors for a dark background. I like 4 spaces for my indenting, no tabs. I use the F5 key to toggle search result highlighting, and F6 to toggle between paste mode (no auto-indent) and no-paste mode.

set backupcopy=yes
set softtabstop=4
set autoindent
set cindent
set shiftwidth=4
set shiftround
set expandtab
set formatoptions=t
set textwidth=0
set mouse=a
set backspace=indent,eol,start
set ignorecase
set smartcase
set incsearch

map <f5> :set hls!<bar>set hls?<CR>
map <f6> :set paste!<bar>set paste?<CR>

let &t_ti.="\e[1 q"
let &t_SI.="\e[5 q"
let &t_EI.="\e[1 q"
let &t_te.="\e[0 q"

syntax enable
set background=dark

nmap rr :%s/\s\+$//g

let &tags=$CTAGS_DB

My mom used to make this Pozole when I was a kid, and I really enjoyed it. Good strong flavors with fun accessories at serving time. Good use of tomatillos.

Pozole

  • 2 T. cooking oil
  • 2 pound pork roast – Trimmed and cut into 1 1/2″ cubes
  • 1 large onion, minced
  • 4 garlic cloves, minced
  • 5 Cups chicken stock
  • 10 tomatillos, husked, cored, and quartered
  • 1 bunch fresh cilantro, stemmed (de-stemmed?)
  • 1 15oz. can hominy, drained and rinsed
  • 2 dried red chilies
  • Salt, pepper
  • Garnish:
    • 1 med onion finely diced
    • Shredded lettuce
    • Thinly sliced radishes
    • Lime wedges

Heat oil in large heavy skillet (Dutch oven) over high heat. Add pork and cook until brown on all sides.

Reduce heat to medium. Add large onion and cook until tender stirring occasionally, about 10 minutes.

Add garlic and cook 2 min.

Add 3 cups stock. Simmer until meat is very tender, about 90 minutes.

Puree tomatillos and cilantro with remaining 2 cups stock in blender. Add puree, hominy, and red chilies to pork. Simmer 15 min. Season with salt and pepper.

(Can be prepared 2 days ahead. Cover and refrigerate. Rewarm over low heat)

Discard Chilies. Serve pozole, passing diced onion, lettuce, radishes and line separately.

A friend was looking for a way to list the space usage on a windows server that only had FTP access. I had written something similar for a project long ago, and polished up to do the job.

This python script will walk an FTP directory in a top-down, depth-first pattern. It uses the ftplib library, which I believe is built-in to most or all python distributions. Configure the FTP_* variables near the top to set the server, port, user, password, and the delay between each FTP operation (to avoid hammering the server). The script recursively processes directories, creating a dirStruct tuple that contains the following items:

(pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)
    pwd is a string like "/debian/dists/experimental"
    subdirList is a list of tuples just like this one
    fileList is a list of (filename, sizeInBytes) tuples
    sizeInFilesHere is a sum of all the files in this directory
    sizeTotal is a sum of all the files in this directory and all subdirectories

It also writes data to two CSV files:

  • dirStruct_only_folders.csv
    • Contains entries for just the directories.
    • Local size is the total size of files in that folder (does not count subdirs).
    • Total size is the sum of local size and total size of all subdirs.
  • dirStruct_complete.csv
    • Contains entries for both files and folders.
    • Files do not have a total size, only a local size.
#!/usr/bin/env python
#
# A script to recursively walk an FTP server directory structure, recording information
# about the file and directory sizes as it traverses the folders.
#
# Stores output in two CSV files:
#  dirStruct_only_folders.csv
#     Contains entries for just the directories.
#     Local size is the total size of files in that folder (does not count subdirs).
#     Total size is the sum of local size and total size of all subdirs.
#  dirStruct_complete.csv
#     Contains entries for both files and folders.
#     Files do not have a total size, only a local size.
#
# Customize the FTP_* variables below.
#
# Basically does a depth-first search.
#
# Written by Matthew L Beckler, matthew at mbeckler dot org.
# Released into the public domain, do whatever you like with this.
# Email me if you like the script or have suggestions to improve it.

from ftplib import FTP
from time import sleep


FTP_SERVER = "ftp.debian.org"
FTP_PORT = "21" # 21 is the default
FTP_USER = "" # leave empty for anon FTP server
FTP_PASS = ""
FTP_DELAY = 1 # how long to wait between calls to the ftp server

def parseListLine(line):
   # Files look like          "-rw-r--r--    1 1176     1176       176158 Mar 30 01:52 README.mirrors.html"
   # Directories look like    "drwxr-sr-x   15 1176     1176         4096 Feb 15 09:22 dists"
   # Returns (name, isDir, sizeBytes)
   items = line.split()
   return (items[8], items[0][0] == "d", int(items[4]))

# Since the silly ftp library makes us use a callback to handle each line of text from the server,
# we have a global lines buffer. Clear the buffer variable before doing each call.
lines = []
def appendLine(line):
   global lines
   lines.append(line)
def getListingParsed(ftp):
   """ This is a sensible interface to the silly line getting system. Returns a copy of the directory listing, parsed. """
   global lines
   lines = []
   ftp.dir(appendLine)
   myLines = lines[:]
   parsedLines = map(parseListLine, myLines)
   return parsedLines
   
def descendDirectories(ftp):
   # Will return a tuple for the current ftp directory, like this:
   # (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)
   #     pwd is a string like "/debian/dists/experimental"
   #     subdirList is a list of tuples just like this one
   #     fileList is a list of (filename, sizeInBytes) tuples
   #     sizeInFilesHere is a sum of all the files in this directory
   #     sizeTotal is a sum of all the files in this directory and all subdirectories

   sleep(FTP_DELAY) # be a nice client

   # make our directory structure to return
   pwd = ftp.pwd()
   subdirList = []
   fileList = []
   sizeInFilesHere = 0
   sizeTotal = 0

   print pwd + "/"
   items = getListingParsed(ftp)
   for name, isDir, sizeBytes in items:
      if not isDir:
         fileList.append( (name, sizeBytes) )
         sizeInFilesHere += sizeBytes
      else:
         # is a directory, so recurse
         ftp.cwd(name)
         struct = descendDirectories(ftp)
         ftp.cwd("..")
         subdirList.append(struct)
         sizeTotal += struct[4]

   # add in the size of all files here to sizeTotal
   sizeTotal += sizeInFilesHere
   return (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)

def pprintBytes(b):
   """ Pretty prints a number of bytes with a proper suffix, like K, M, G, T. """
   suffixes = ["", "K", "M", "G", "T", "?"]
   ix = 0
   while (b > 1024):
      b /= 1024.0
      ix += 1
   s = suffixes[min(len(suffixes) - 1, ix)]
   if int(b) == b:
      return "%d%s" % (b, s)
   else:
      return "%.1f%s" % (b, s)

def pprintDirStruct(dirStruct):
   """ Pretty print the directory structure. RECURSIVE FUNCTION! """
   print "{}/ ({} in {} files here, {} total)".format(dirStruct[0], pprintBytes(dirStruct[3]), len(dirStruct[2]), pprintBytes(dirStruct[4]))
   for ds in dirStruct[1]:
      pprintDirStruct(ds)

def saveDirStructToCSV(dirStruct, fid, includeFiles):
   """ Save the directory structure to a CSV file. RECURSIVE FUNCTION! """
   # Info about this directory itself
   fid.write("\"{}/\",{},{}\n".format(dirStruct[0], dirStruct[3], dirStruct[4]))
   pwd = dirStruct[0]

   # Info about files here
   if includeFiles:
      for name, size in dirStruct[2]:
         fid.write("\"{}\",{},\n".format(pwd + "/" + name, size))

   # Info about dirs here, recurse
   for ds in dirStruct[1]:
      saveDirStructToCSV(ds, fid, includeFiles)

print "Connecting to FTP server '%s' port %s..." % (FTP_SERVER, FTP_PORT)
ftp = FTP()
ftp.connect(FTP_SERVER, FTP_PORT)
if FTP_USER == "":
   ftp.login()
else:
   ftp.login(FTP_USER, FTP_PASS)

print "Walking directory structure..."
dirStruct = descendDirectories(ftp)

print ""
print "Finished descending directories, here is the info:"
pprintDirStruct(dirStruct)
print ""

FILENAME = "dirStruct_complete.csv"
print "Saving complete directory info (files and folders) to a CSV file: '%s'" % FILENAME
with open(FILENAME, "w") as fid:
   fid.write("\"Path\",\"Local size\",\"Total size\"\n")
   saveDirStructToCSV(dirStruct, fid, includeFiles=True)

FILENAME = "dirStruct_only_folders.csv"
print "Saving directory info (only folders) to a CSV file: '%s'" % FILENAME
with open(FILENAME, "w") as fid:
   fid.write("\"Path\",\"Local size\",\"Total size\"\n")
   saveDirStructToCSV(dirStruct, fid, includeFiles=False)

Sample CSV output:

"Path","Local size","Total size"
"/plugins/",5426535,7594527
"/plugins/foo-1.1.jar",7774,
"/plugins/CHANGELOG.txt",45169,

Local size is just the size of the file itself, or the size of all files in a directory. Total size is the total size of the files in a directory plus the total sizes of all subdirectories. Files do not have a total size entry.

I recently discovered that two of the hard drives in my server had a firmware bug that could leave to silent dataloss when used with smartd (and other programs). Fortunately, there is a firmware update. Unfortunately, it doesn’t change the drive’s reported firmware revision so you can’t tell if the update has been applied already…

To run the update, it says “Save the .exe files to a bootable media”. It doesn’t provide any more details than that, but apparently it needs to be a DOS boot disk/usb drive. This link provides an easy way to take a pre-generated FreeDos image and DD it to a USB drive. Once you do that, you can mount it and copy the files to run to the FAT partition.

FreeDOS prebuilt bootable USB flash drive image – http://chtaube.eu/computers/freedos/bootable-usb/

As part of migrating to a new account at Dreamhost, I had to regenerate my blog from an old SQL dump backup email. Not quite as easy as from an actual WordPress export, but I made it work. However, something got messed up with the RSS feed. Turns out my old blog install had the following in the /blog/.htaccess that didn’t make it into the new install:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /blog/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]
</IfModule>

Adding it to the same place in the new install fixed things, but changed the location to https://www.mbeckler.org/blog/?feed=rss2 . Just an FYI.

I’m writing a fun little webapp using Flask and Python and Sqlalchemy, running on Heroku using a PostgreSQL database. I use a sqlite3 database file for local testing, and PostgreSQL when I deploy, so naturally there are some minor snags to be run into when switching between database engines.

Tonight I ran into a tricky issue after adding a ton more foreign-key / relationships to my database-backed models. I was getting an error like this when I tried to issue my db.drop_all() command in my python script that initializes my database tables:

sqlalchemy.exc.InternalError: (InternalError) cannot drop table pages because other objects depend on it
DETAIL:  constraint pagesections_parent_page_id_fkey on table pagesections depends on table pages
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
 '\nDROP TABLE pages' {}

A bunch of searching for solutions indicated that maybe it would work if you run db.reflect() immediately before the db.drop_all(), but apparently the reflect function is broken for the current flask/sqlalchemy combination. Further searching revealed a mystical “DropEverything” function, and I finally found a copy here. I had to do a few small modifications to get it to work in the context of Flask’s use of Sqlalchemy.

def db_DropEverything(db):
    # From http://www.sqlalchemy.org/trac/wiki/UsageRecipes/DropEverything

    conn=db.engine.connect()

    # the transaction only applies if the DB supports
    # transactional DDL, i.e. Postgresql, MS SQL Server
    trans = conn.begin()

    inspector = reflection.Inspector.from_engine(db.engine)

    # gather all data first before dropping anything.
    # some DBs lock after things have been dropped in 
    # a transaction.
    metadata = MetaData()

    tbs = []
    all_fks = []

    for table_name in inspector.get_table_names():
        fks = []
        for fk in inspector.get_foreign_keys(table_name):
            if not fk['name']:
                continue
            fks.append(
                ForeignKeyConstraint((),(),name=fk['name'])
                )
        t = Table(table_name,metadata,*fks)
        tbs.append(t)
        all_fks.extend(fks)

    for fkc in all_fks:
        conn.execute(DropConstraint(fkc))

    for table in tbs:
        conn.execute(DropTable(table))

    trans.commit()

I had to change the uses of engine to db.engine since Flask’s SQLalchemy takes care of that for you. You get the db object from the app, like this “from myapp import db”, and this is how I defined db in myapp:

from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__, etc)

# DATABASE_URL is set if we are running on Heroku
if 'DATABASE_URL' in os.environ:
    app.config['HEROKU'] = True
    app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL']
else:
    app.config['HEROKU'] = False
    app.config['SQLALCHEMY_DATABASE_URI'] = "sqlite:///" + os.path.join(PROJECT_ROOT, "../app.db")

db = SQLAlchemy(app)

And then this is the important parts of my db_create.py script:

from sqlalchemy.engine import reflection
from sqlalchemy.schema import (
        MetaData,
        Table,
        DropTable,
        ForeignKeyConstraint,
        DropConstraint,
        )

from cyosa import app, db

if not app.config['HEROKU'] and os.path.exists("app.db"):
    os.remove("app.db")

def db_DropEverything(db):
    # listed above

db_DropEverything(db)
db.create_all()

# add your instances of models here, be sure to db.session.commit()

I wanted to play Minecraft on my 64-bit Ubuntu Linux install, but it wasn’t working correctly for me, and would give a black screen after login, and the console reported some errors about xrandr (which might be related to my odd “dual display-port + docking station” setup at home). After some searching, I found a tip to manually install the LWJGL java libraries into the ~/.minecraft/bin/ folder, to have the latest and greatest version of those libraries.

  1. Download the latest version zip archive of the LWJGL libraries: http://sourceforge.net/projects/java-game-lib/files/latest/download?source=files
  2. Extract downloaded zip archive
  3. Copy all files in lwjgl-2.9/jar/ to ~/.minecraft/bin/
  4. Copy all files in lwjgl-2.9/native/linux/ to ~/.minecraft/bin/natives/

And then you should be good to go.

via https://bbs.archlinux.org/viewtopic.php?pid=876274#p876274

Every stinkin’ time you have to update Java (which is every day it seems, since Java has more (security) holes than most Swiss cheese) it wants to install the stupid Ask Toolbar and take over your default search engine. Here’s a quick Windows registry fix that will apparently disable the installer from even asking you about the toolbar installation.

Another way, without having to download and rename or create a new .REG file, is to copy and paste the following two lines into an elevated CMD prompt:

reg add HKLM\software\javasoft /v "SPONSORS" /t REG_SZ /d "DISABLE" /f

reg add HKLM\SOFTWARE\Wow6432Node\JavaSoft /v "SPONSORS" /t REG_SZ /d "DISABLE" /f

via Superuser: How can I prevent Ask.com Toolbar from being installed every time Java is updated?