History Meme 0

Posted by timgoh
on Thursday, April 24

Just an “I’m still here, just busy” type post. Originally from Mark Pilgrim

I’ve had this Thinkpad for 3 months, which means I run vim approximately once an hour (this may seem abnormally low, but once I’m inside I use :tabf and :tabe to open more files).


blackwater ~: history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head -n 20
1976 v
1937 cd
1545 ls
1390 svn
883 rake
832 ssi
471 grep
400 sudo
383 cap
349 rm
292 cat
217 ssh
209 ./script/server
209 cp
208 mv
181 riff
179 cdiff
175 scp
168 psql
147 screen

Key

v = gvim -p
ssi = svn stat --ignore-externals
riff = ssh to slice
cdiff = svn diff $* --diff-cmd colordiff

Vim Users, cure your Textmate envy

Posted by timgoh
on Monday, April 07

My exposure to Textmate was all “whoa this looks sweet, but I can’t give up my Vim key bindings and plug-ins”. Fortunately there’s ways to accomplish the look of most Textmate installs and not have to switch away from the best editor out there.

Get Monaco font for Linux

  1. Download the Monaco Linux truetype font Monaco_Linux.ttf
  2. Find out what your font directories are:
    ~: cat /etc/fonts/fonts.conf | grep directory -A5
  3. Create a new directory under one of your font directories and move the Monaco_Linux.ttf into it. For my distro /usr/share/fonts/truetype works:
    # mkdir /usr/share/fonts/truetype/custom
    # mv Monaco_Linux.ttf /usr/share/fonts/truetype/custom/
  4. Refresh your font caches:
    # fc-cache -f -v
  5. If you’re using vanilla Vim, you may have to restart your console application of choice, then change your console font. If you’re using GVim, add the line
    :set guifont=Monaco\ 10
    Replace ‘10’ by whatever font size you would like to use. Do note that there is a space after the backslash (ie the space is escaped).

Get Vibrantink or Vividchalk color scheme

Vibrantink link

Vividchalk link

Jo Vermeulen (contributed vibrantink) has a helpful post with a screenshot comparing the original Textmate theme to Vibrantink and Vividchalk.

Put the .vim file you download in your .vim/colors directory, and you can then add the line

:colorscheme vividchalk
to your .vimrc file.

I personally prefer vividchalk myself.

If everything worked out, you now have the ‘bling’ of Textmate with the functionality of Vim. A winning combination, if you ask me!

Migrating to FeedBurner

Posted by timgoh
on Sunday, April 06

This is long overdue.

Threw up a 301 from the old address at http://www.progprog.com/feeds/atom.xml to http://feeds.feedburner.com/progprog . Please change your addresses to the FeedBurner feed from now on.

Rails won't be mainstream, but that's fine by me

Posted by timgoh
on Friday, March 21

First post!

For the company blog that is.

With my luck, my byline will be truncated in the wrong place by search engines, and searches for my name would turn up “charged multiple times [dot dot dot]” instead of “charged multiple times with method_missing abuse”.

Use plus-addressing with your Gmail address for filtering

Posted by timgoh
on Thursday, March 20

A lot of people know this already, but I’ve seen a few blank faces when I mention it in real life1, so I thought it’s worth a mention.

The gist of it is, joeschmoe@gmail.com, joeschmoe+purchases@gmail.com, joeschmoe+XYZ@gmail.com will all be received by the address joeschmoe@gmail.com, where XYZ is anything alphanumeric. This is very useful because you can filter on the “To” address, which makes it very specific.

So you can use joeschmoe+paypal@gmail.com for your Paypal account, and have a 100% accurate filter for any e-mail from Paypal, no matter what its subject, contents, or From address is. Plus if they ever sell your address to spammers, you would know.

There are lots of possibilities, so try what works out for you. You can have a specific joeschmoe+amazon alias, or register for multiple online stores with a joeschmoe+purchases alias.

If you do this with a brand new e-mail address and never give out the base local-part, you can constantly maintain a zero message inbox, with every incoming mail perfectly filed.

One caveat: not all sites obey RFC2822 to the letter – some of them disallow plus signs in the local-part (I suspect that most times this is not intentional, just incompetence—since it is much easier to copy-paste the canonical e-mail regex than roll your own).

PS. If there really is someone out there using joeschmoe@gmail.com… Mr. Schmoe you have my profound apologies.


1 You know how it is: you’re in a crowded bar, and some beautiful girl comes up to you and goes, “Excuse me, would you happen to know a more efficient way of filtering my e-mail?” while licking her lips slowly.

Career Update

Posted by timgoh
on Thursday, March 20

Well the first two months of 2008 were certainly interesting. Lots of jumping through hoops and paperwork. But the outcome of all that is, I’m now in LA and officially an employee of CitrusByte, a kick-ass web app development firm. My main requirement in looking for jobs is working on fun projects with smart people daily, and CitrusByte certainly scores very high in this regard1.

Being in such an overachieving environment has also made me get off my ass, and I have a bunch of personal projects in the pipeline that I’ll be releasing “any time now”(tm). So do keep an eye out.


1 And no they didn’t pay/force me to write this

Introducing Open Source to high school kids

Posted by timgoh
on Monday, March 17

I had the chance two weeks ago to give a talk to kids at my alma mater. These aren’t ordinary kids—they’re precocious kids in the Computer Elective Program which means they do C++ in 9th grade1 and PHP+MySQL in 10th grade.

In this post I’ll be outlining the motivation and content of my talk, and also what I’ve learned about talking to that age group, in the hope that other people with a similar opportunity can benefit.

Although it was titled otherwise, my talk was essentially “What I wished someone had told me when I was in high school”. Back then I was pretty much a Microsoft and Borland fanboy due to complete ignorance of open source. I had started with Basic and Pascal and the switch to Windows had really thrown me for a loop. All of a sudden coding had become so much more difficult. Visual Basic 1.0 eased the pain for a while, but ultimately I did not enjoy developing with it.

With just about all apps going the Windows route, I got disillusioned with coding for a while due to the tediousness of the Windows API and stopped for a while. I did use Turbo Pascal to train for algorithm coding competitions but didn’t do any small app development.

If someone had told me about open source back then… gotten me to take up Python or some other high-level dynamic language and mess around with the code of high quality open source projects… I don’t think my coding would have stagnated for that period of time. I just wish someone had told me there was an alternative that would make programming fun, and most importantly, responsive! A teenager does not enjoy the endless boilerplate that was the original Windows 3.0 and 3.1 API.

Since I can’t change my past, I decided to talk to these students and let them know more about open source. The state of OSS presently is much less hostile to newbies than it used to be. There’s easy-to-use distributions like Ubuntu, lots of advances in hardware compatibility, and very viable alternatives to PC software that didn’t exist during my time.

“During my time”. That’s one phrase I consciously avoided during my talk. I’m already more than a decade older than my audience—why separate myself even further? When I was in school, any similar phrase always alienated me from the speaker. It emphasizes the age gap even further. Don’t use that phrase.

Here’s a brief outline of what I talked about. I’m not providing slides because it’d be pretty pointless—a lot of them have just one word or picture. I’m not a run-of-the-mill Powerpoint bullet point abuser!

  • who am i
  • why be a software developer
  • brainteasers
  • why open source
  • fun projects to look at
  • why school projects are unrealistic
  • how to improve your coding skills
  • starting an open source career while still in school

Ok, so here’s my little self-evaluation

What went well:

  • I was able to hold the attention of most of my audience. A couple were sleeping, but everyone else pretty much paid attention. Since my primary goal was simply awareness of open source, I am pretty satisfied with this. A couple of them had even tried Ubuntu already but no one used it as a primary OS.
  • The brainteasers near the beginning of the talk got people thinking, and involved in the subsequent talk. (Get your audience to think through something at the beginning of a talk, and they’ll be more committed to the rest of your session than if you had just kept speaking throughout). I had two “what does this do” type questions – one which was a long series of piped shell commands, and another a Ruby fragment.
  • I managed to emphasize how cool it is to go through the applications and see what’s going on. I did a quick run-through of Amarok and then opened up its code in KDevelop. That was pretty exciting.

What didn’t go so well:

  • I may have been able to hold their attention during the talk, but there hasn’t been much response since then. I deliberately chose not to list any resources. I simply made my e-mail available and told them to contact me with specific requests and interest, so that I could cater my recommendations accordingly. Only 3 people came to talk to me after my session in search of pointers, and I have not heard from anyone since.
  • I did not go into detail about any topic. While time constraint did have something to do with this, in retrospect I think a “let’s code a scraper on the spot right now” would have been more interesting, and I could have just trimmed certain details from other parts of the presentation.

So the final verdict is, I did what I set out to do, but I could have done it better.

I found the entire process very rewarding. I really urge people to give it a shot—it doesn’t have to be the high school you went to. Find a high school with a computer club, and try and arrange to give a talk there. Contributing to Open Source isn’t about coding. Make a small time investment in winning the minds of future coders to Open Source.


1 I highly highly disagree with this curriculum. A language like Python, Ruby, or heck even C# would be much better at teaching concepts. C++ is inferior as a language to be taught because students will have to spend more time on syntax rules than learning algorithms.

Singapore is hostile to software developers

Posted by timgoh
on Friday, March 07

Basically the island is blanketed in transparent proxies (at least 2 of the major ISPs use them) with no WebDAV support. Which means in the default case, you cannot run ‘svn up’, ‘svn checkout’, etc. You’d have to set up an alternate proxy for subversion to use.

I’m surprised that googling didn’t turn up anyone complaining about this. Does no one in Singapore use Subversion? Some of my friends who are developers here say that they do all their development remotely. There is such a high pain factor to working over SSH/VPN/VNC/etc 100% of the time that I wonder why they have to do this.

It’s strange how a country that is supposedly so high-tech is so bass ackwards when it comes to something so straightforward.

Colorize your shell

Posted by timgoh
on Tuesday, January 22

We all love syntax highlighting, but often don’t bring that love with us to our shell. Here’s a couple tips to have a useful multi-colored shell session

Shell Prompt

Add

export PS1='\[\033[1;34m\]\h\[\033[0;33m\] \w\[\033[00m\]: '

to your bashrc.

This will give you your host in blue and working directory in brown. One excellent thing to do with this is to color-code each of the servers you log in to regularly differently. That way you have an instant, easy to understand indicator of which server you are currently accessing, so that you don’t ‘rm -rf’ from the wrong shell.

To customize, the Red Hat Knowledgebase lists more options for colors, and will come in handy for the rest of this post as well.

If you want to make sense of that command, Linux.com has a quick tutorial

Searches

Came across this tip from scie.nti.st

Add

export GREP_OPTIONS='--color=auto'
export GREP_COLOR='1;33'

to your bashrc, and have matches highlighted.

Or of course you could use tools like ack or rak which have colorization built-in and are more efficient than the venerable ‘grep’.

Man pages

Get the ‘most’ command1. On Debian and derivatives it would be a straightforward

sudo apt-get install most
.

You should already have a

export PAGER=less
or similar line in your bashrc file. Change that to
export PAGER=most
Now ‘man’ a command, and find out what you’ve been missing.

Background

One of the new things I’ll be doing with ProgProg this year is writing on various practices I have accumulated over the years. I wouldn’t go so far to call them “best practices”—they are just things that work for me.

My previous company had a great habit of not ever sitting on its laurels when it came to process. We were constantly tweaking, and occasionally making significant changes to our process in an attempt to improve. I have adopted this concept for myself, and once a week I examine my work habits and pick out something I can do in a better way. I’m not promising a weekly ‘process’ post, but will be presenting anything I find extremely useful.


1 “Most” is a misnomer though. Given that thanks to geek humor “less” is better than “more”, the command should really be called “least”.

My Dream for a Slice

Posted by timgoh
on Monday, January 21

Alright, I’m back. Not quite completely settled in Los Angeles yet, but I’m getting there.

ProgProg will be a whole new blog in 2008. Don’t believe me?

Just the infrastructure alone:

  1. Typo -> Mephisto
  2. Dreamhost -> Slicehost

The second change is not in response to their latest billing fiasco (although it did shock me because I was planning to leave anyway and I suddenly got stuck with a huge bill). It’s just that I haven’t needed that much hand-holding on the server side for a long time, and the increased flexibility of my own VPS appealed to me.

Topic matter will change as well. As I mentioned in my previous post, I won’t be writing too much about Python and Django anymore (I do have a post brewing on Django, but it won’t be a “Django is a Ghetto” entry1).

Once again apologies for the break – I’m back and good to go.


1 See here if you don’t get the reference.

Hiatus 1

Posted by timgoh
on Sunday, December 02
I won't be posting any new articles for a while since I will be relocating over the next month. Also I won't be using Django at my new job, so I may not be blogging about Django as often. So this is me signing off for 2007.

Thoughts on the Django Sprint 0

Posted by timgoh
on Sunday, December 02

The Django sprint took place on December 1st, and while I couldn’t commit too much time I did manage to add a patch for one ticket, for making 403 pages customizable instead of being hard-coded.

It wasn’t an extremely difficult ticket since there were ample examples of how 404 and 500 pages enabled customization. I paired with my colleague Ionut on the ticket and we went about it pretty comprehensively imho, with tests for our changes, and also added tests for the CSRF middleware (previously there were none).

One minor issue with the sprint was the server outage for the Django Trac site that lasted a few hours. Surely someone had to know about this before hand (the hosting company gave 10 days notice) and could have done anything from simply posting about the outage on the sprint wiki page to making arrangements for the page to be mirrored.

Also it might have been more efficient to divvy out tickets before the sprint itself, instead of a “come and grab what you’d like to do” approach. I understand that the core devs don’t want to be too heavy-handed in managing the project, but I think a worldwide sprint could deal with a tad more organization.

These minor grievances aside I think the sprint went really well. Having events like this where everyone gets together (virtually or physically) is a good way to bring new developers into a project. At my office, we had several participants unfamiliar with Django itself, let alone Django internals. But by the end of the sprint they left with some knowledge of Django, and the entire workflow behind contributing to it.

More views on the Django sprint from Bjorn and Ionut

Now Microsoft's Facebook stake makes sense 0

Posted by timgoh
on Sunday, November 04

So, Microsoft spent $240 million on 1.6% of Facebook (valuing it as the 5th largest Internet company). Then Google launches OpenSocial, with big names like MySpace, Six Apart (MovableType and LiveJournal), etc.

Is it any surprise that Microsoft ended up investing in the closed proprietary platform – one which will compete vs an open platform that’s essentially “everyone else that matters”?

I can just imagine it…

The suits of each of the companies involved in OpenSocial are discussing potential partners. Someone pipes up, “Hey you know what? Let’s get Microsoft to join us. They’ll love the idea of an open social network, and would contribute a lot to getting this project off the ground.” Everyone cracks up laughing, and the guy who spoke can no longer keep a straight face. “Okay okay, seriously, who else should we approach?”

Shared Del.icio.us Accounts with Python 2

Posted by timgoh
on Wednesday, October 31

syncdelish is a little script I wrote that takes links in a del.icio.us user’s ‘links for you’ section and adds them to his bookmarks.

Basically if you run this script on crontab you automatically bookmark any links users have tagged as “for:”. You can use this to provide link-sharing via a single communal account for a group of people, or simply to auto-add links people send your way.

To use this for a shared account, just create a new account ie “deli”. Have everyone add this account to their del.icio.us network, and then just tag links you want to share with “for:deli”. Those links will be added to deli’s bookmarks whenever the script is run, so you will be able to access them easily1.

Script and usage

Download the script here (BSD licensed2)

Rename as you like and run with Python. Usage is:
syncdelish USERNAME PASSWORD FOR_YOU_FEED

Where FOR_YOU_FEED is the link to RSS at the bottom of the ‘links for you’ page. It looks something like ‘http://del.icio.us/rss/for/username?private=[longhash]’.

Known Issues

‘Links for you’ seems to be an incomplete del.icio.us feature. There are limitations with it as follows:

  • these links cannot be deleted by you
  • even if the original user who tagged it removes the for: tag, it still is not removed
  • there is no differentiation in the private feed for saved links and unsaved links
  • the private ‘links for you’ feed is protected via security by obscurity – no authentication needed, just relies on people not guessing the correct hashcode.

Also, the del.icio.us posting API has only two return codes: success and failure. There is no notification of “failure because duplicate”. Hence when a previously saved link is encountered, the script wrongly outputs failure. It doesn’t break, it just mistakenly tells you that this link could not be added instead of saying it already existed.

Basically, currently it seems that unless you are a del.icio.us developer you have no way of removing something tagged as for you. This leads to an entire more serious problem…

It means my script gets slower every time a link is added, since the list of links to copy only grows and never shrinks. Once the number of shared links gets to a reasonable size, the script is all but unusable, since my script does not know which is a duplicate.

There are a few ways I can resolve this:

  1. Scrape from HTML instead of XML (the link after toggles between ‘saved’ and ‘save this’). Rejected because that’s not the right way to do it.
  2. Log processed links in a file and skip over them if encountered. Rejected because one of my chosen constraints for this was implementing it with a single script.
  3. Take a ‘last_time_run’ or ‘frequency’ type optional parameter that skips all links added before the time given. This is still not ideal because the full list of for: links still has to be downloaded and processed. In other words it’s a hack. So, rejected for now, but if the status quo remains I will add in this feature.
  4. Whine to del.icio.us feedback about the limitations with ‘for you’ links, and then write a blog post whining some more.

As you already know, I picked option 4 for now. Do check back on this page for any updates.

Background and Credits

Thanks to Simon Willison I came across Natalie Downe’s Snafflr script. I liked the idea, but I wanted to implement my own version using del.icio.us’s native “for:” tag.


1 Links tagged with “for:” can only be accessed by the user account they are shared with.

2 Only reason I chose BSD over MIT is that the former has the name of my alma mater while the latter was too expensive for me to attend.

Using database views with Django models 0

Posted by timgoh
on Wednesday, October 17

I’m surprised this feature isn’t promoted more (maybe because it involves raw SQL?). You can hook a Django model to a database view. It can come in very useful when you’re trying to aggregate common information from various models.

Problem to solve

Say I have the following models: Album, Artist, and Dvd on a music site that accepts submissions. Before submissions are shown on the site they must be approved. So each of them has a boolean saying whether they are approved or not, something like:

approved = models.BooleanField(default=False)

Let’s complicate the situation a little to make for a better example. Say, the equivalent field for the Dvd model is called “not_unapproved1”. Also, Albums and Dvds have a foreign key to record label2, something like:

record_label = models.ForeignKey(EvilRIAAMember)

Our goal is to have a model called Unapproved which aggregates the Album, Artist, Dvd objects that have not been approved. It should provide access to the ‘approved’ and ‘record_label’ fields. Then I can placate my site editors who are clamoring for an easy way to find all unapproved content regardless of content type.

Step 1: View definition

UNAPPROVED_VIEW = """ 
CREATE OR REPLACE VIEW unapproved_view AS
  SELECT
    (SELECT nextval('unapproved_sequence')) as id,
    o.id AS object_id,
    ct.id AS content_type_id,
    o.approved AS approved,
    o.record_label_id AS record_label_id
  FROM
    album as o,
    (SELECT id FROM django_content_type where model='album') as ct
  WHERE
    NOT o.approved
UNION
  SELECT
    (SELECT nextval('unapproved_sequence')) as id,
    o.id AS object_id,
    ct.id AS content_type_id,
    o.not_unapproved AS approved,
    o.record_label_id AS record_label_id
  FROM
    dvd as o,
    (SELECT id FROM django_content_type where model='dvd') as ct
  WHERE
    NOT o.not_unapproved
UNION
  SELECT
    (SELECT nextval('unapproved_sequence')) as id,
    o.id AS object_id,
    ct.id AS content_type_id,
    o.approved AS approved,
    NULL AS record_label_id
  FROM
    artist as o,
    (SELECT id FROM django_content_type where model='artist') AS ct
  WHERE
    NOT o.approved
""" 

Put that definition in app/models/unapproved.py, which is where the model will be going as well.

Those familiar with db views should probably skip on ahead to the next section.

Ok, what this gives you is a view called ‘unapproved_view’ which has the following fields:

  • id
  • object_id
  • content_type_id
  • approved
  • record_label_id

Since artists do not have a record label foreign key, we specifically NULL it out. Note that we are only selecting rows that are not approved. If you’re trying this out with your own models, remember that for a UNION to work you need to have matching types.

The “nextval(‘unapproved_sequence’)” part is just to give each row in the unapproved_view a unique ID.

You may be tempted to optimize by replacing (SELECT id FROM django_content_type where model='artist') AS ct with a hard-coded content_type_id value, but that is much less safe. I’d rather have the overhead of those extra queries and not have to worry about content type ids changing on me.

Step 2: Model File

In app/models/unapproved.py:

  class Unapproved(models.Model):
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()
    approved = models.BooleanField(default=false)
    record_label = models.ForeignKey(EvilRIAAMember, null=True)

    class Meta:
      db_table = "unapproved_view" 

This isn’t the whole file of course, fill in the rest: imports, app_label, etc.

Things to note:
  • this model has “null=True” for the record_label definition to cater for Artists not having one.
  • 0.96 and below should use models.GenericForeignKey instead of generic.GenericForeignKey

Anyway you now have a model file hooked to the view. You’ll have to create that view in your database first, of course (run the contents of the UNAPPROVED_VIEW variable from Step 1). Don’t forget about creating the sequence:

CREATE SEQUENCE unapproved_sequence START 1;

Step 3: Use the darn thing already

Now if you add the necessary “class Admin” boilerplate to the model, you’ll find that you can access this model in the Django admin. While you can’t view individual objects and they’re read-only, you do get a handy list view.

You can use this model in (Django) views and templates though. You can now do things like Unapproved.objects.filter(...). And with the GenericForeignKey you have easy access to the actual content object. Writing a view that displays all unapproved objects and lets user change the “approved” value to True is trivial once you have this set up, so I won’t get the details of that.

Hope the above explanation helped. This is a very useful technique since it allows you to practice DRY and handle common properties and behavior across your models in a single place.


1 Turns out the coding for Dvd model was outsourced to someone whose native language favors double negatives.

2 Yes, wise guys, in real life artists have record labels too, but since these can possibly change over time, for the purposes of this demo they don’t!