Roy's Techblog

Quitting Facebook is all the rage right now. I appear to have hopped on a bandwagon unintentionally by initiating the deletion of my Facebook account.

I joined Facebook late, in March 2008, primarily because I needed to understand the basics of Facebook app development for work. I did spend a little bit of time exploring the social graph component and other aspects relevant to my employer, but beyond that I rarely used it other than to accept or reject friends.

I was a little curious to explore a bit as it was the first time/place online I'd run into so many non-technical people from my past in an online situation.

So why should I shut down my account given I use it so little?

Here's my reasons:

• I've always been wary about Facebook, and have been on the verge of deleting my account since mid 2009 when Facebook was found to violate canadian privacy law.
• Given facebook's increasing privacy problems, I just don't feel comfortable allowing them to continue to share any of my data (including my social graph).
• Their ever expanding public sphere bothers me. A lot.
• Very few people use Facebook to do anything but add me as a friend.
• I'm becoming more and more concerned about maintaining my privacy in an online environment where privacy is seemingly scarce.
• I work with the Facebook API regularly at work, and have a very good idea what information you can easily access.  It bothers me.

I was having doubts about whether or not I should let my deletion proceed, given it seems to have become trendy to delete (I don't like to be trendy)… Then something happened that helped me be sure I'd done the right thing:

Today at work, Facebook prompted a colleague to add a second email address. The popup correctly identified five possible past or present email addresses of his.  No incorrect addresses were suggested.  Some of the email addresses suggested were over 5 years old.

This kind of thing makes me nervous in the hands of any company.

Farewell, Facebook.

Anyone who wishes to contact me may do so by email, SMS, telephone, IM, selected private IRC channels, or one MOO.

 

 

I've always felt that the ubiquitous windowed model of computing with a single pointing device is inefficient. It seems that the folks at 10/GUI agree, and so does Lukas Mathis and others like Scott at ISO50

I'm excited by R. Clayton Miller's 10/GUI, primarily because it is a step in the right direction. Reducing the reliance on haphazardly placed windows and the introduction of a single multi-purpose and very powerful input device are great ideas. For a long time, I've been thinking on and off about how to get out of the windowed paradigm. I've never succeeded.

I dislike the mouse - its clumsy, inaccurate, and hurts. I like keyboard controls, but my fingers get confused beyond two meta keys at a time. I love my multitouch touchpad on my MacBook. I also love the multitouch on the iPhone, but find its screen frustratingly small and the ergonomics for lengthy use to be poor. The obstruction problem mentioned in the 10/GUI video constantly occurs.

All that said, I don't feel that the ideas presented in 10/GUI address power-users, software developers, or multi-document/multi-monitor use very well. I don't even think it deals with normal business multi-tasking very well either.

Frequently, a business user will require two or three applications at a time:

• Communication tools (possibly two) such as MSN/iChat/GTalk/Skype, etc.
• Email
• Reference materials (documents, browsers, help files, emails(s), customer service database screens, administrative tools, etc.)
• Working task (eg. a new document, a new email, administrative tools, etc.)

I believe that a better solution would be to be able to break the workspace into zones and affix "windows" to these zones. Zones could be associated with portions of each display.

I believe that this type of arrangement would help users effectively use increasingly common large and wide-screen monitors, while still performing tasks that require reference materials.

This arrangement should scale well to software developers and power users with dual (or more) monitors, too:

Things that might exist in the Optional Dashboard Widgets would be weather, stocks, news/rss headlines, email alerts, calendar previews, parcel tracking.

Things that might be in the Communications Tool(s) section include VOIP, Instant Messaging, Email status, RSS feeds, Microblogging, etc.

My feeling is that by being semi-windowed in a structured manner, people will be able to continue to use their computers in a more familiar way. This should ease the transition to a new window management philosophy while remaining somewhat familiar. This would allow users to continue to perform their tasks in a familiar way in a novel GUI, thus easing transition and likelihood of success.

Today, a number of tweets showed up for twitviewer.net.  I haven’t been able to load that site, so I can’t comment on what its about, but it reminded me of something I recently helped with at work.

There is no need for application developers who integrated with Twitter to ask for your password any more.  Just say no!  If you encounter a site or software that asks for your twitter username and password, refer them to the Sign in with Twitter documentation.

Sign in with Twitter uses OAuth, "An open protocol to allow secure API authorization in a simpleand standard method from desktop and web applications."  The main thing to note about OAuth this is that you authorize applications to use Twitter at twitter.com instead of within the application/website.  

Here at work, we use Twitter’s OAuth to allow users post a status update when they make a new travel blog entry.  Here’s a screenshot of the authorization process:

First I click "Sign in with Twitter":

   

Then I click Allow:

That’s it!  I get returned to the site I was on before, and its authorized to interact with my account as described above.

For a desktop application, the process differs slightly.

 

WIth Adium 1.4 beta, you start by initiating the process in your desktop program, which opens a browser.  Here’s the first two steps:

Next, you get shown an access code that you need to type into the desktop application to complete the process.  Tthe one you’re seeing here is no longer valid.

 

 

Another important thing is you can also revoke access without having to change your password, and authorized applications remain authorized even if you change your password.

 

So go convince your favorite website or twitter application to switch now!

 

Something that I've fought with for a long time is how to make good nightly off-site backups of MySQL and PostgreSQL. Having recently switched back to my favourite *nix for server use, FreeBSD, I am now using ZFS on my server. (Boy is it fast. But that's a post for some other time.)

One of the advantages to using ZFS is that it supports the notion of serialising/deserializing file-systems or snapshots so you can transport them elsewhere and restore them, possibly on the same system to a new pool. One such use is remote backups of databases.

In the past, I've tried nightly full dumps using mysqldump or pg_dump and then rsyncing them along with the rest of my data for nightly backups. This meant that I had an ever-growing quantity of data to ship. Other ideas I had included using diff to generate incremental deltas which could be sequentially patched to generate the latest database dump. This approach could work well, and would only require two full dumps to be kept around (today's and yesterday's)... but I never got around to it. And it would still require periodic full dumps to be copied over, since you really don't want to rely on trusting that many patches will apply cleanly. Plus, depending on your workload, the diffs could be as big as or bigger than the full dump, since diff is line-based. Any modification to one column in every row of a big database would result in a huge delta - for example, adding a new column.

Back to ZFS. My idea is simple: Use Ralf S. Engelschall's snapshot tools for FreeBSD, to keep about a week's worth of nightly snapshots (plus some weekly ones), and send only the nightly ones to a remote site every night. The trick with remote backups is that there's a chance that a night will be missed. ZFS doesn't do all the necessary magic to determine if snapshots were renamed and to determine which ones aren't yet applied.

That's where my simple little sync_remote_snapshots.pl script comes in. This little script is intended to run from the receiver ("local"). It uses ssh as the transport, something you'll have to setup to work without passphrases while the script is running (ideally using ssh-agent).

The script starts by listing the local and remote zfs snapshots (using zfs list -t snapshot -r zpool/fsname), parses the output, then matches up the local and remote snapshots by timestamp. This is done because the snapshot numbers won't match up between the two machines after the next snapshot is run. This is because the freebsd snapshot script renumbers the snapshots such that @daily.0 is the lowest numbered snapshot for a daily snapshot. If two days go by, then @daily.0 on the local end would be for two days prior.

In order to be able to run the incremental backup, we need to first rename all the local snapshots just like the remote side does, purge any old local snapshots we don't want any more, and then actually do the backup.

This script takes care of all that. It also handles the case where the snapshot failed mid-way (loss of network, out of space, etc), leaving it such that the local snapshot numbers don't start at zero.

There's definitely situations it doesn't cover, but it does what I need and uses way less bandwidth than rsyncing compressed nightly dumps.

While on a call with my carrier, changing to newer voice and data plans to save a bit of cash, this error popped up on my iPhone: “UICC Update failed 2” with accept or decline options at the bottom. After reading about the Universal Integrated Circuit Card at WikiPedia, it became apparent this error must have been related to changing my service options. The error happened twice, once for each edit the customer service representative made.

Twice now, I’ve helped friends recover lost photos from flash cards. Both friends are technical people who generally know what they’re doing … but they both got bitten by bugs in commercial software that left them without their photos for one reason or another.

The loss of photos is a very upsetting event to a photographer. Fortunately, if you stop using the card the instant you discover the problem, more than likely, your photos can be recovered. If you never use the on-camera delete to remove anything but the most recently taken photo(s), you’re in even better shape, since no fragmentation will have happened!

So how do you recover photos when there’s NO filesystem information left? Turns out its pretty easy for some file formats and tools that ignore trailing junk in files. The approach is simple: snapshot the raw card contents and then look for the start-of-file signature for your specific type of photo files— but most importantly, look for it aligned at the start of 512-byte boundaries —the typical “block” size.

To determine your file’s start signature, you’ll need a few sample files from your camera. If you have a Canon Rebel XTi and shoot in RAW mode, you’ll be getting CR2 files. These files have a nice long signature at the start, making detection and recovery a breeze.

To determine if there’s a good unique signature, you can do something like this:

for f in IMG*.CR2; do hexdump -n 16 $f; done | sort | uniq -c
     6 0000000 4949 002a 0010 0000 5243 0002 3006 0001
     6 0000010

If there were 6 files in the directory, and you only get 2 lines of output, you’ve found yourself a reliable 16-byte signature. More than enough to detect the start of files in most cases, especially when aligned to the start of a 512-byte block.

The above signature is what’s needed for a CR2 file.

To obtain the necessary image of the flash card (I don’t recommend ever working directly on the flash card when doing recovery—so we read it once and save it for future processing).

So how do you grab the contents of the flash for safe processing? Under Linux, FreeBSD and OSX (and other unix platforms), you use dd. We do this because we don’t want any extra headers… just the raw bytes. This ensures the disk remains aligned at 512-byte boundaries. Some disk image containers might happen to keep things aligned to the block boundaries. I’ve never checked.

The specific instructions for OSX are as follows:

1. Using a card reader, mount the flash card (like usual: just insert it)
2. Start Terminal.app and run df to find the device name of the newly mounted flash card. We’re specifically interested in the /dev/diskNsN device name, since we’re going to need to directly access it. Here’s an example:

   /dev/disk4s1      999344    978464     20880    98%    /Volumes/EOS_DIGITAL

3. Next, we need to unmount the disk without causing the device to be removed. We can either use unmount /dev/disk4s1 or go to Disk Utility, select the right volume, then use the “Unmount” toolbar icon to unmount it without ejecting it.
4. Finally, we create the image we’re going to work with using dd.

   dd if=/dev/disk4s1 of=flashimage.dat

   Depending on the size, speed of the card, your card reader, and USB interface, this could take a long time. If you need to know how far it’s gotten, open a new Terminal window and run “du -h flashimage.dat”

Once the copy of the image has been made, we’ll want to run a quick recovery script. This script relies on the fact that Canon Raw conversion utilities tend to ignore trailing junk. If yours don’t, grab http://cybercom.net/~dcoffin/dcraw/:”dcraw” (available via http://www.macports.org/:”Mac Ports”) and convert the files to a format you can use (like TIFF).

Here’s the recovery script I hacked together to recover missing CR2 files for my friend:

#!/usr/bin/perl

use strict;
use warnings;

my @signature = qw(49 49 2a 00 10 00 00 00 43 52 02 00 06 30 01 00);
my $signature = pack("H*", join("", @signature));
my $siglen = length($signature);
my $blocksize = 512;

open(IN, "flashimage.dat") || die "$!";

# Skip 2gb to get past existing files
seek(IN, 2 * 1024 * 1024 * 1024, 0);

my $block = "";
my $imgno = 0;
while (read(IN, $block, $blocksize)) {
    if (substr($block, 0, $siglen) eq $signature) {
        print "starting $imgno\n";
        open(OUT, sprintf(">found%04d.CR2", ++$imgno));
    }
    print OUT $block unless !$imgno;
}

The end result should be a whole bunch of .CR2 files named from found0001.CR2 through the final number.

PS – The above should work for JPEG files, but JPEG headers aren’t as big/consistent. The above technique has been proven to work for a Pentax *ist with a fully erase card (unerase couldn’t be performed) before any files were written and a Canon Rebel XTi where 1/2 the card had been filled back up with new photos. Some files might be corrupt due to fragmentation. Basically, what I’m saying is: the only two cameras I know of so far that write their files in a sane manner without fragmentation (unless holes are created by erasing files on the camera) are the above two cameras.

Unfortunately, out of the box, the postgres gem fails to install with the MacPorts postgres82-server install. There are two problems. The first is it can’t find pg_config. The second is incorrect architecture detection (thanks to Andreas Flierl at the RubyForge postgres module forums for pointing out the fix).

ERROR:  Error installing postgres:
    ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb install postgres
extconf.rb:73: command not found: pg_config --bindir

The fix for this one is to fix the path temporarily:

 export PATH=/opt/local/lib/postgresql82/bin/:$PATH

Then when we retry the gem installation, we’ll get the following error(s):

postgres.c:41: error: static declaration of ‘PQserverVersion’ follows non-static declaration
/opt/local/include/postgresql82/libpq-fe.h:262: error: previous declaration of ‘PQserverVersion’ was here
postgres.c:41: error: static declaration of ‘PQserverVersion’ follows non-static declaration
/opt/local/include/postgresql82/libpq-fe.h:262: error: previous declaration of ‘PQserverVersion’ was here
postgres.c: In function ‘Init_postgres’:
postgres.c:2676: error: ‘pgconn_protocol_version’ undeclared (first use in this function)
postgres.c:2676: error: (Each undeclared identifier is reported only once
postgres.c:2676: error: for each function it appears in.)
postgres.c:2677: error: ‘pgconn_server_version’ undeclared (first use in this function)
postgres.c: In function ‘Init_postgres’:
postgres.c:2676: error: ‘pgconn_protocol_version’ undeclared (first use in this function)
postgres.c:2676: error: (Each undeclared identifier is reported only once
postgres.c:2676: error: for each function it appears in.)
postgres.c:2677: error: ‘pgconn_server_version’ undeclared (first use in this function)
lipo: can't open input file: /var/tmp//ccBatpen.out (No such file or directory)
make: *** [postgres.o] Error 1

These are fixed by specifying a specific architecture to use.

The following commands worked well for me:

export PATH=/opt/local/lib/postgresql82/bin/:$PATH
sudo env ARCHFLAGS="-arch i386" gem install --remote postgres

After installing the Mac Ports postgresql82-server port, I tried to make it run. It didn’t. The problem was that su didn’t like running anything as user postgres. Here’s the original instructions:

###########################################################
# A startup item has been generated that will aid in
# starting postgresql82-server with launchd. It is disabled
# by default. Execute the following command to start it,
# and to cause it to launch at startup:
#
# sudo launchctl load -w  /Library/LaunchDaemons/org.macports.postgresql82-server.plist 
###########################################################  
--->  Installing postgresql82-server 8.2.6_0

To create a database instance, after install do
 sudo mkdir -p /opt/local/var/db/postgresql82/defaultdb
 sudo chown postgres:postgres  /opt/local/var/db/postgresql82/defaultdb 
 sudo su postgres -c '/opt/local/lib/postgresql82/bin/initdb -D  /opt/local/var/db/postgresql82/defaultdb'

And here’s my test to determine if su was working:

 sudo su postgres -c id

I got no output… So I tried

 sudo -u postgres id

And I get:

 uid=401(postgres) gid=401(postgres) groups=401(postgres)

So now what?

There’s an easy way to deal with this: Grab the server admin tools from Apple and install them.

Next, fire up Workgroup Manager (its in your /Applications/Server folder). Ignore the dialog and go straight to the menu and select Server -> View Directories. Select the PostgreSQL Server user in the left-hand pane, then select the Advanced tab and change the Login Shell to something like /bin/sh. See the picture for details. Don’t forget to hit Save.

Now the command from before works:

# sudo su postgres -c id
uid=401(postgres) gid=401(postgres) groups=401(postgres)

So now we can finish the last step to initialize the database, then fire up Postgres.

  sudo su postgres -c '/opt/local/lib/postgresql82/bin/initdb -D /opt/local/var/db/postgresql82/defaultdb'

If you already ran the launchctl command, you can make Postgres start with these two commands:

 sudo launchctl unload -w /Library/LaunchDaemons/org.macports.postgresql82-server.plist
 sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql82-server.plist

Otherwise just run the load version of the command.

Picking a good bag seems to be hard to do. It’s possible I’m just fussy, but I do have some particular needs.

An ideal bag must carry:

1. a 15” or so laptop
2. a textbook or novel
3. a moleskine notebook (or two)
4. miscellaneous papers like bills, printed academic papers, a magazine, or printouts needed for working at home
5. a couple CDs or DVDs (in paper covers)
6. a few pens/pencils in an external pouch
7. some business cards
8. a power adapter
9. various cables (usb, serial, ipod adapters)
10. full sized headphones when travelling
11. an iPod 5G
12. a camera the size of a Canon G7

And it must:

1. be rain resistant
2. be padded
3. be comfortable to carry when fully loaded

And finally:

1. I’d like it not to squash my papers.

So far, the candidates that have potential bags are:

1. Crumpler
2. Tom Bihn
3. Timbuk2
4. BOOQ
5. and I had been pondering Courierware
6. and Manhattan Portage

One of the problems I have had is deciding whether I want a backpack or shoulder-bag of some form (such as a messenger bag). You’d think this would be an easy choice, but it turns out not to be.

I currently carry a Brenthaven Pro 15/17 for work, but it belongs to work, and I’m changing jobs. This bag is great, but its not quite ideal. The laptop compartment takes up a very large percent of this bag, and while the design lets you overstuff it, I find I quite often damage papers or books. There really isn’t that much space to carry papers around, although the divided paper compartment is a great thing. The top pocket on the front of it for pencils and other little things is wonderful, and the top-loading zipped up slot for the laptop is great, but I always worry that it won’t hold out in a downpour. The bottom cable compartment is also great, and the front flap with space for your miscellaneous stuff is great… But its just not quite ideal for me because things get crumpled too easily.

A friend carries around a 2-3 year old Tom Bihn Brain Bag and loves it, but I’m not sure if it fully addresses what I need. It certainly has some nice pockets and the Freudian Slip looks like it’ll help avoid crumpled papers. I can’t remember if it had a top pocket that I’ve grown to love on the Brenthaven.

I went and had a look at a Timbuk2 Classic Messenger at a local shop in both the Large and Medium sizes, to get an idea for how the bags would be in their Laptop Messenger line. The Large is frickin’ huge—perhaps too big. I’d probably damage everythig in my bag all the time. It just doesn’t have enough organizational aids. The Medium looks much better size-wise, and had a nice amount of open space… But I think I’d need both the laptop compartment and a divider. I’ve also looked at people’s photos of what they carry in their Commute at Flickr to see if I think it’d do. Sadly, I think its just a little too small.

Maybe the Super Ego is what I need?

I’ll post again when I finally make up my mind.

Update: Since originally writing this article, a friend suggested I look at Chrome as well.

Update: I finally picked the Tom Bihn Brain Bag brain bag with a Freudian Slip and a Snake Charmer (for when traveling) and an appropriately sized Horizontal Brain Cell which holds the laptop snugly and with plenty of padding. I’ve taken to using the Brain Cell to haul the laptop around the house. I’ll post a review of my bag soon.

One thing that has significantly changed since 10.4 is that NetInfo is gone. To create users, you now have to use dscl, and the syntax is quite different than with nictl.

Here’s what I did to add a user for AMANDA, a backup program:

sudo dscl localhost -create /Local/Default/Users/amanda
sudo dscl localhost -create /Local/Default/Users/amanda RecordName amanda
sudo dscl localhost -create /Local/Default/Users/amanda UserShell /bin/bash
sudo dscl localhost -create /Local/Default/Users/amanda RealName "Backup User" 
sudo dscl localhost -create /Local/Default/Users/amanda UniqueID 5000
sudo dscl localhost -create /Local/Default/Users/amanda PrimaryGroupID 0
sudo dscl localhost -create /Local/Default/Users/amanda NFSHomeDirectory /Users/amanda