Nov 05

The last job on Earth, part 1

I see that the “conventional wisdom” these days is that programmers will be obsolete once we figure out how to describe programs in a more human way.

I think that’s absurd.  It would take only one glance at the volumes upon volumes of federal laws or state laws to demonstrate just how incredibly difficult and byzantine “human accessible” rules are.

Indeed, I suspect that if computers & software didn’t exist, most programmers would be lawyers (except for the ones that would be mathematicians) because trying to understand and follow laws and loopholes and exceptions and conditions of laws is equivalently difficult to understanding code.



Aug 27

10 months of no inbox zero

I’ve been running “inbox zero” since about the year 2000.     Up until 10 months ago, when someone emailed me, asking me for a copy of the sequel to my first book.    And then, just a few weeks later, a second guy emailed me, asking for the same thing.

As a motivator to help me finish, I kept those two emails unread.  So I’ve been at “Inbox 2” for the last 10 months.

I’m pleased to say that as of last night, I am back to inbox zero.  *Whew*



Dec 10

Using WiKID Two-Factor Authentication with SSH

I have found a couple of howtos on how to set up WiKID to use SSH, but I had a horrible, horrible time using them.  This is partially because of my version of Linux (RHEL 5 and 6) and partially because the howtos don’t do a good job of explaining why you are taking various steps.   I want to remedy that issue, so that others can learn from my pain.


Background information

Most of you probably know that on Linux, at least, ssh uses PAM to authenticate users.  PAM is a pluggable tool that allows you to control how different applications authenticate users when they try to log into a machine.   By default, typically, PAM will provide instructions and rules that ensure that users are authenticated against the /etc/passwd database.

But you can include other tools in the authentication system (and in fact, you can write you own).   One of these PAM rule libraries is focused on connecting PAM to RADIUS – an authentication protocol that was very popular back in the dial-up modem days.     It continues to live on in various implementations, including, most popularly, FreeRadius.


FreeRadius was built many years ago, and relatively recently, a new version was created: FreeRadius2.     Because of the relatively glacial pace of Linux software change (not necessarily a bad thing), many of the documents and howtos still reference FreeRadius, instead of FreeRadius2.   More frustratingly, FreeRadius2 is sometimes labelled as simply “FreeRadius”, which can be confusing.

FreeRadius and PAM

FreeRadius includes a module, specifically built to help integrate RADIUS with PAM.   That would be the aptly-named PAM Authentication and Accounting Module.  Now, as far as I can tell, it only comes as source, so be prepared to build it. (I hope you know how to use/install Make and cc/gcc, etc).

In addition, you can’t just download it and build it, you need to install the FreeRadius base RPM, as well as the developer and utility RPMs.   After that, you should have a which you can install into /lib/security or /lib64/security, depending on your OS.

This module is fairly straightforward – it attempts to find a configuration file called /etc/raddb/server, and looks in the file, to determine which RADIUS server to authenticate against.   It then sends that server UDP packets on port 1812, and if there’s a RADIUS server listening there, the server can answer and validate the username/password provided.

Updating PAM’s sshd configuration

On my system, I have to update the /etc/pam.d/sshd file in order to include the .  I found three different examples of how to do this on the Internet, and all three were different, so I won’t pretend that I know the “right” answer here.  Let me just say that I put my entry fairly early in the config file, which seemed like the correct thing to do.  On my system:

auth	   sufficient  debug

Is the very first line.   debug is there so I could get some logging output of what was happening, to help troubleshoot.  Once you’re live, remove it.

The situation so far

At this point, if you’ve followed the steps  described above, when you log in with ssh, it will attempt to find a RADIUS server and authenticate against it.

“Well, I have just installed FreeRadius, why don’t I use that” your instincts might say, at this point.  But your instincts are wrong.

In my odyssey to get this software working, I have discovered that, at the end of the day, once you have the compiled, and you have configuration rules specified in /etc/raddb/server, you could (if you wanted) safely delete FreeRadius.  You don’t need it anymore.  Keep the utility and dev libraries though.  You’ll need those later.

Setting up /etc/raddb/server

Again, as I mentioned earlier, is preconfigured to look for a configuration file named /etc/raddb/server.  This file contains the following line:	your_shared_secret       3

What does this do?   Well, it tells that library ( that the RADIUS server it wants to talk to is running on localhost, on port 1812 (by default), and it “knows” the shared secret key “your_shared_secret“.

Completely unnecessary test

If you had a wild hare, or you just like billing by the hour, at this point, you could set up FreeRadius, add a fake user to /etc/raddb/users, and see if you could log in as that user, with the username and cleartext password specified in the /etc/raddb/users file, and the shared secret “your_shared_secret” specified in the /etc/raddb/client.conf  . There are already guides for doing this, if you feel compelled.

Will it work for ssh at this point?

No, not until you add that user to /etc/passwd.  LFMF!

Ok, done.  Now will it work for ssh?

Probably.  But that’s not what you want.  You want to use WiKID, right?  (otherwise, why are you here?).   That’s a bunch of additional steps along the path.

Ok, so how do I set it up with WiKID?

Well, first off, install WiKID.   Set up the server using their documentation.   Note that if you try to move it off of ports 80 and 443, you will be miserable.   LFMF!

When it comes time to set up a Network Client, set up a RADIUS client, using “your_shared_secret“.


When you do this, make sure that FreeRadius isn’t running.  Also, make sure that you restart WiKID after you add the new Network Client.

Basically, when you add a RADIUS network client to WiKID, WiKID becomes a RADIUS server!. I cannot stress this enough – it took me almost two weeks to realize that I was trying to connect a server to another server.  Yes, I am dumb.   LFMF!

Ok, I have a network client

Well, now you need to add a WiKID user.  This was relatively straightforward from the documentation, except for one little detail – when you install the wikid software token on a laptop, PC, phone, whatever, you have to:

a) Register it back with the WiKID server

b) Make sure that the user exists in /etc/passwd

c) You can’t ever re-use a registration with a given username.  So if you, say, end up testing the client on two different computers, with two different registrations, you have two different users. If you attempt to log in with the client token registered for user A, but with the username for user B, it will silently fail.


You could try ssh at this point, by following these steps:

1) get your wikid temporary passcode

2) ssh with the username associated with the passcode, and use the passcode as the password

3) If you have a shell, congrats!

If you have problems, there are some pretty good logs built in to the WiKID web ui.   Set various things to debug, and you’ll see more info.     Note that these logs disappear after restarting WiKID, so stay vigilant about turning them on.  note also that after a few failures, your users are silently disabled, which will cause logins to silently fail, even if everything else is correct.  Stay vigilant!

Note also that you can use the Freeradius test tool (radtest) to test your configuration, if ssh isn’t cooperating (or if you don’t want to change ssh until the very end).

Testing with radtest

radtest <username> <wikid_passcode> 0 <your_shared_secret>

username: the username in WiKID and also in /etc/passwd

wikid_passcode: the temporary 1 minute passcode you get from using the WiKID token client.

your_shared_secret :  the secret you used to configure the RADIUS Network Client in WiKID.

I can say that the logs you get from using radtest are easier to follow than using ssh directly.


At this point, you should be able to use ssh to log in, using the WiKID username, and the temporary passcode.   Good luck!















Dec 10

True dat

Dec 10

The Fourteen Types of Programmers: Type 7: Random ones

There are really two subsets of programmers here.   First, this is the catchall for everyone who doesn’t fit into any of the other categories.  A bit of a cop-out, but then, it’s been six years since I started this list, so perhaps you want to just cut me some slack maybe 😉


The second subset is real – it is the programmers who change things at random when something isn’t working.

Good Things:

  • There are situations, like debugging or when you’re trying to learn something new, where changing things at random and finding out what happens next can be incredibly illuminating.
  • This technique helps you learn how the software works – it helps you build your mental model (assuming the software is consistent), or it angers and enrages you when it (the software) is inconsistent.

Bad Things

  • Unfortunately, in my experience most Random programmers are people who don’t understand what they are doing, and change things at random in production systems, to see what works.
  • And when they find anything that works, they stop.
  • Which leads to nasty, horrible, confusing, mismatched code – pretty much the worst of all possible “successful” worlds.
  • Big Ball of Mud (the famous anti-pattern) is almost certainly a by-product of Random programming
  • Which implies that there are a _lot_ of them out there.

How to identify them

  • The Good ones say things like:
    • “I picked up this new library/framework/app/etc – and I’m trying to build something with it to learn how to use it”
    • “I discovered, through trial and error, that if you do X, Y happens.  After figuring out why, I was able to solve the problem in a consistent way”
  • The Bad ones say things like:
    • “I found some code that seemed to solve a similar problem, so I copied it.”
    • “I discovered, through trial and error, that if you do X, Y happens.  So it’s fixed.”



Nov 21

freeradius2 and pam

just a quick rant while i build up steam on this particular issue.

Why oh why is this so ridiculously hard? I can find lots of advice on how to get freeradius1 integrated with pam, but the naming conventions and the shared libraries are different in FR2, and for the life of me, I can’t find any indication of which FR2 library I need to add to PAM so it (PAM) can use FR2 to authenticate a user. BLEAH!

Nov 02

Just let it crash? No thanks

This blog post discusses the philosophy of not trying to catch every exception – let the app crash when you don’t understand or anticipate the problem

I agree that you shouldn’t just catch Exceptions, politely log them and then swallow them.    You then place the burden on yourself (or the operations guys) to actually realize that there’s a problem – the customer may not know – they may just think the app is horrible.   Without operational feedback, you’ll never know that the app is misbehaving.

Here’s what I do – in the situations where an unexpected Exception is thrown, I capture the exception, log it, send a polite error message to the user explaining that something went wrong, and then I email the stack trace to myself.

That way, I know when things have gone wrong.   This has saved me a dozen times – errors I didn’t know about, issues with disk space, issues with an edge condition that I thought was practically impossible.

So, my advice: don’t ignore the exceptions, don’t “just let it crash”.

instead: Recover, and behind the scenes, sound the alarm.




Nov 02

OpenVPN – very well done

After my struggles with OpenLDAP, I have to admit – OpenVPN’s commercially packaged version (which is free to set up and use for the first 2 simultaneous users) is incredibly easy to get up and running.   And additional users are $50/10, which is chump change.

Sep 29

OpenLdap: History’s greatest monster

Well, not really, but I bet I got your attention.


Having just gone through two days of pain involved in getting OpenLDAP to work in place of /etc/passwd, I have such a set of troubleshooting tips unlike anything I’ve experienced before.   I am going to write these down as quickly as I can, before the horrifying experience is mercifully driven from my mind.

Note that this was using RHEL 6.0 and yum

  • Recent (as in 2012, perhaps 2011) versions of OpenLdap use a different config structure than the older version.  Dramatically different.   Many of the examples I found on the ‘net are based on the old model, and can be very, very confusing.
  • Lots of examples discuss creating your own database of stuff.  But you dont have to do that!  There’s already a database ready and waiting for you in: /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
    • Update that DB, instead of creating your own bdb, which is what most of the examples do.
  • I had several problems getting syslog to properly capture the OpenLdap logging events.   I had to add an olcLogFile= directive to actually get things to log.
  • This was pretty clear, but just for the sake of thoroughness, I had to add a ‘top level’ user who could do most management functions.  This was very helpful with creating users, setting passwords, etc.  I created:
    • olcRootDN: cn=Manager,dc=example,dc=com
    • olcRootPW: {SSHA}Cw888Xm4M77777Zb8Zhut7LLgwHfb3y8b
    • in the bdb database (described above).
      • use slappasswd to generate the {SSHA} password from clear text.
  • The examples of rules to set up ACLs so that users can access their own data, and change their own password were woefully unclear.   I kept getting this error in the logs: “RESULT tag=103 err=50 text=” .  That’s because the ACLs weren’t set up right.
  • To set up the ACLs so your users can access their own data, and change their own password (but no one else can even see it), here were the magical lines I had to include in: /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
    • olcAccess: {0}to attrs=userPassword by self write by anonymous auth by dn.base=”cn=Manager,dc=example,dc=com” write by * none
    • olcAccess: {1}to * by self write by dn.base=”cn=Manager,dc=example,dc=com” write by users read by * none
  • I had to install all of the following packages to successfully get ssh to connect to OpenLdap
    • pam_ldap
    • openssh-server
    • openssh-clients
    • openssh-ldap
    • openssh
    • openldap-clients
    • openldap-servers
    • openldap
    • nss-tools
    • nss-sysinit
    • nss
    • nss-util
    • nspr
  • I had to modify the following config files:
    • /etc/openldap/ldap.conf
    • /etc/openldap/slapd.d/cn=config.ldif
    • (the aformentioned bdb database, which is where the various users are stored.  more about that later)
    • /etc/pam_ldap.conf
    • /etc/nsswitch.conf
    • /etc/nslcd.conf
    • /etc/sysconfig/authconfig
    • /etc/sysconfig/ldap
    • /etc/ssh/sshd_config
  • I used authconfig to enable ldap in PAM.  Wait, that sounds really dirty.
  • I had to copy /usr/share/openldap-servers/DB_CONFIG.example into /var/lib/ldap/DB_CONFIG to get the database to stop complaining about not having a configuration
  • I had to define the People and Group organizational units, and use them consistently.
  • Learn how to use ldapadd to add data to the database.  Clunky, but easier than adding it by hand.

Here’s what I used to create my People and Group organizational units:


dn: dc=example,dc=com
objectclass: dcObject
objectclass: organization
dc: example

dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
objectClass: top
ou: People

dn: ou=Groups,dc=example,dc=com
objectClass: organizationalUnit
objectClass: top
ou: Groups

After that, I could add users to both the People and Group domains:

Adding a user to People:

dn: uid=jbsw,ou=People,dc=example,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: John SW
uid: jbsw
uidNumber: 1001
gidNumber: 1001
homeDirectory: /home/jbsw
loginShell: /bin/bash
userPassword: {crypt}x
shadowLastChange: 0
shadowMax: 0
shadowWarning: 0

I took that data, wrote it to a file called user_jbsw.ldif, and then imported it using ldapadd:

ldapadd -x -D "cn=Manager,dc=example,dc=com" -f /etc/openldap/user_jbsw.ldif -W
and to group:
dn: cn=jbsw,ou=Groups,dc=example,dc=com
objectClass: top
objectClass: posixGroup
cn: jbsw
userPassword: {crypt}x
gidNumber: 3001

I wrote that data to a file called group_jbsw.ldif

ldapadd -x -D "cn=Manager,dc=example,dc=com" -f /etc/openldap/group_jbsw.ldif -W

And last but not least, (for that user) I used ldappasswd to change that password to something that ldap and ssh would accept:

ldappasswd -s blarneystone -D "cn=Manager,dc=example,dc=com"  -W -x uid=bsw,ou=People,dc=example,dc=com

Then what happened?

I had everything set up properly – I had pam_ldap.conf pointing to the LDAP server, the LDAP server was running, sshd was configured to use PAM, but I still couldn’t log in.  It kept complaining about unauthorized access (error 49).   Eventually, I realized that I hadn’t installed the nss package(s), which created yet another way to describe how to talk to ldap.

And then all was well?

Well, no, after I figured that out, and got the password working, it demanded a new password.  I tried to enter it, but it wouldn’t accept it, saying I didn’t have the rights to modify the user object in LDAP.  That was when I figured out the ACL issue (already described above).

And then you were done?

Well, no, in the process of monkeying around with /etc/pam.d/system-auth, I inadvertantly set up the ‘change password’ functionaity so it asked for and verified the new password twice.   But that was just a matter of editing.

Anything else?

  • modifications to  /etc/ssh/sshd_config require a restart of the sshd daemon
  • modifications to /etc/nslcd.conf require a restart of the nslcd daemon
  • modifications to openldap config require a restart of slapd
    • I also deleted all the data in /var/lib/ldap/ each time I modified /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
  • modifications to PAM were instantaneous.

Hope all my pain helps someone, somewhere.



Jun 12

Signalling vs. Hyperbolic discounting

Bryan Caplan has a lot of thoughts on signaling as the motivator for education.   The idea makes a lot of sense, up to a point.   But let’s  go back and re-assemble my college mental model, as best I can.

So I sign up for a class, and I pay my class’s $1000 tuition up front.  It’s now a $1000 sunk cost.   Let’s say there will be 50 hours of classroom lecture between now and the end of the semester.    Let’s also say, for argument’s sake, that the class is something topical to my expected career path.

And due to illness, the first class is cancelled.  Woo hoo.  Yes, I’ve lost 2% of my class time, but I am quite sure that I can still learn 100% of what I need to learn (because a lot of class is fluff or bookkeeping) in 98% of the time.

The second class is cancelled.  Still happy about it.

Third class is cancelled.  Okay, first week, nothing.  I’m probably still happy, but starting to get a little concerned.

Fourth class is cancelled.  Arg.  Fifth. Arg.   Sixth?  Um, really?   Seventh, Eight, Ninth – ok, at this point, I’m annoyed, and I start complaining.

So, this seems useful – after about 20% of the class has been cancelled, I start to feel a little cheated.   I paid to learn something useful and potentially interesting, and I’m not getting my money’s worth.   If it was all about signaling, I wouldn’t have cared – I would have happily gotten an ‘A’ and never gone to class at all – no one would really be the wiser.

What makes more sense to me is that I accept that the class isn’t a perfectly efficient mechanism for skill transfer, and that a modest amount of “slack” is not going to meaningfully impact what I learn.

Another way to think of it is that the knowledge I gain from a class is not homogenous – not every skill, not every fact is of equal value.  And it’s reasonable to assume that if the class is cancelled, that what “falls off the table” in terms of skills learned are the least valuable skills and the least useful facts – the most critical skills and facts are still taught, just a day later.

Or, to wrap it up in economese:  the marginal cost of a single cancelled class is low, based on the rational expectation that the professor will adjust her schedule and ensure that the most valuable skills are still taught.




Jun 16

Hadoop competitor

Interesting – if it works and deploys well, HPCC is a good thing for the scalable processing space.

Apr 19

Favorite Portal 2 quote so far


Okay, look. We both said a lot of things that you are going to regret. But I think we can put our differences behind us. For science… you monster


Do you know the biggest lesson I learned from what you did? We’re a lot alike, you and I. You tested me, I tested you. You killed me, I – oh no, wait. I guess I haven’t killed you yet. Well … food for thought

Apr 12

Android app in 1.5 hours

I have to give great props to the Google Android team. I was able to go from “I want to build an app for my phone” to “My app is now published on the Android market” in approximately 90 minutes of effort.

1) Download the SDK – that was easy.

2) Add the plugin to Eclipse – also easy

3) Set up the various platforms in the SDK “system”. This was confusing at first. Eventually I realized that one wants (typically) to go for ‘lowest common denominator’ – so Android API 1.5

4) Find a skeletal app to rework. I was writing a soundboard, so it turned out to be incredibly easy to repurpose.

5) Make my app changes – easy, because of the template.

6) Load the emulator and try out my changes. This took a little while to get right, because I didn’t have the target set up properly in my app.
7) Run the emulator and test my code – straightforward

8) Create a package-able version, and load it directly onto my phone

9) Use Astro file manager to install the app from the phone itself – easy

10) Verify that the app worked on my phone – easy. surprisingly so.

11) Create a signed package for publishing (this took a little while because the wizard was a little confusing, but not horrible)

12) Sign up as an android developer. This cost me $25, but was very easy.

13) Upload my app. This took a little while because I had to rename it (there was a conflict), create screenshots, etc.

14) Publish.

Within minutes, the app was live on the Android Marketplace.

Feb 16

Startups and Agile Teams

I have spent a long time working in software startups of various stripes, and I have also spent a long time working with Agile teams delivering software of various stripes.

It is  difficult to sufficiently emphasize how similar those two models are.    Think about it:

Startup Agile
What’s the simplest thing that could possibly sell? Working Software over Comprehensive Documentation
Hire smart, talented people, and most of the problems will take care of themselves Individuals and interactions over processes and tools
Get something out quick so your customers can give you feedback Customer collaboration over contract negotiation
What you end up building is almost never what you originally envisioned Responding to change over following a plan

Read this fascinating article about Dropbox, and how it succeeded where its competitors did not. It should help you down this path.

Jan 04

random Jim Croce-inspired madness

If I could save Time in a Timesheet
if Jobs could make iPods call you
Google would cache every page til Eternity passes
and then it would trend them for you….

Older posts «

» Newer posts