Roy's Techblog

Twice now, I’ve helped friends recover lost photos from flash cards. Both friends are technical people who generally know what they’re doing … but they both got bitten by bugs in commercial software that left them without their photos for one reason or another.

The loss of photos is a very upsetting event to a photographer. Fortunately, if you stop using the card the instant you discover the problem, more than likely, your photos can be recovered. If you never use the on-camera delete to remove anything but the most recently taken photo(s), you’re in even better shape, since no fragmentation will have happened!

So how do you recover photos when there’s NO filesystem information left? Turns out its pretty easy for some file formats and tools that ignore trailing junk in files. The approach is simple: snapshot the raw card contents and then look for the start-of-file signature for your specific type of photo files— but most importantly, look for it aligned at the start of 512-byte boundaries —the typical “block” size.

To determine your file’s start signature, you’ll need a few sample files from your camera. If you have a Canon Rebel XTi and shoot in RAW mode, you’ll be getting CR2 files. These files have a nice long signature at the start, making detection and recovery a breeze.

To determine if there’s a good unique signature, you can do something like this:

for f in IMG*.CR2; do hexdump -n 16 $f; done | sort | uniq -c
     6 0000000 4949 002a 0010 0000 5243 0002 3006 0001
     6 0000010

If there were 6 files in the directory, and you only get 2 lines of output, you’ve found yourself a reliable 16-byte signature. More than enough to detect the start of files in most cases, especially when aligned to the start of a 512-byte block.

The above signature is what’s needed for a CR2 file.

To obtain the necessary image of the flash card (I don’t recommend ever working directly on the flash card when doing recovery—so we read it once and save it for future processing).

So how do you grab the contents of the flash for safe processing? Under Linux, FreeBSD and OSX (and other unix platforms), you use dd. We do this because we don’t want any extra headers… just the raw bytes. This ensures the disk remains aligned at 512-byte boundaries. Some disk image containers might happen to keep things aligned to the block boundaries. I’ve never checked.

The specific instructions for OSX are as follows:

1. Using a card reader, mount the flash card (like usual: just insert it)
2. Start Terminal.app and run df to find the device name of the newly mounted flash card. We’re specifically interested in the /dev/diskNsN device name, since we’re going to need to directly access it. Here’s an example:

   /dev/disk4s1      999344    978464     20880    98%    /Volumes/EOS_DIGITAL

3. Next, we need to unmount the disk without causing the device to be removed. We can either use unmount /dev/disk4s1 or go to Disk Utility, select the right volume, then use the “Unmount” toolbar icon to unmount it without ejecting it.
4. Finally, we create the image we’re going to work with using dd.

   dd if=/dev/disk4s1 of=flashimage.dat

   Depending on the size, speed of the card, your card reader, and USB interface, this could take a long time. If you need to know how far it’s gotten, open a new Terminal window and run “du -h flashimage.dat”

Once the copy of the image has been made, we’ll want to run a quick recovery script. This script relies on the fact that Canon Raw conversion utilities tend to ignore trailing junk. If yours don’t, grab http://cybercom.net/~dcoffin/dcraw/:”dcraw” (available via http://www.macports.org/:”Mac Ports”) and convert the files to a format you can use (like TIFF).

Here’s the recovery script I hacked together to recover missing CR2 files for my friend:

#!/usr/bin/perl

use strict;
use warnings;

my @signature = qw(49 49 2a 00 10 00 00 00 43 52 02 00 06 30 01 00);
my $signature = pack("H*", join("", @signature));
my $siglen = length($signature);
my $blocksize = 512;

open(IN, "flashimage.dat") || die "$!";

# Skip 2gb to get past existing files
seek(IN, 2 * 1024 * 1024 * 1024, 0);

my $block = "";
my $imgno = 0;
while (read(IN, $block, $blocksize)) {
    if (substr($block, 0, $siglen) eq $signature) {
        print "starting $imgno\n";
        open(OUT, sprintf(">found%04d.CR2", ++$imgno));
    }
    print OUT $block unless !$imgno;
}

The end result should be a whole bunch of .CR2 files named from found0001.CR2 through the final number.

PS – The above should work for JPEG files, but JPEG headers aren’t as big/consistent. The above technique has been proven to work for a Pentax *ist with a fully erase card (unerase couldn’t be performed) before any files were written and a Canon Rebel XTi where 1/2 the card had been filled back up with new photos. Some files might be corrupt due to fragmentation. Basically, what I’m saying is: the only two cameras I know of so far that write their files in a sane manner without fragmentation (unless holes are created by erasing files on the camera) are the above two cameras.

Unfortunately, out of the box, the postgres gem fails to install with the MacPorts postgres82-server install. There are two problems. The first is it can’t find pg_config. The second is incorrect architecture detection (thanks to Andreas Flierl at the RubyForge postgres module forums for pointing out the fix).

ERROR:  Error installing postgres:
    ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb install postgres
extconf.rb:73: command not found: pg_config --bindir

The fix for this one is to fix the path temporarily:

 export PATH=/opt/local/lib/postgresql82/bin/:$PATH

Then when we retry the gem installation, we’ll get the following error(s):

postgres.c:41: error: static declaration of ‘PQserverVersion’ follows non-static declaration
/opt/local/include/postgresql82/libpq-fe.h:262: error: previous declaration of ‘PQserverVersion’ was here
postgres.c:41: error: static declaration of ‘PQserverVersion’ follows non-static declaration
/opt/local/include/postgresql82/libpq-fe.h:262: error: previous declaration of ‘PQserverVersion’ was here
postgres.c: In function ‘Init_postgres’:
postgres.c:2676: error: ‘pgconn_protocol_version’ undeclared (first use in this function)
postgres.c:2676: error: (Each undeclared identifier is reported only once
postgres.c:2676: error: for each function it appears in.)
postgres.c:2677: error: ‘pgconn_server_version’ undeclared (first use in this function)
postgres.c: In function ‘Init_postgres’:
postgres.c:2676: error: ‘pgconn_protocol_version’ undeclared (first use in this function)
postgres.c:2676: error: (Each undeclared identifier is reported only once
postgres.c:2676: error: for each function it appears in.)
postgres.c:2677: error: ‘pgconn_server_version’ undeclared (first use in this function)
lipo: can't open input file: /var/tmp//ccBatpen.out (No such file or directory)
make: *** [postgres.o] Error 1

These are fixed by specifying a specific architecture to use.

The following commands worked well for me:

export PATH=/opt/local/lib/postgresql82/bin/:$PATH
sudo env ARCHFLAGS="-arch i386" gem install --remote postgres

After installing the Mac Ports postgresql82-server port, I tried to make it run. It didn’t. The problem was that su didn’t like running anything as user postgres. Here’s the original instructions:

###########################################################
# A startup item has been generated that will aid in
# starting postgresql82-server with launchd. It is disabled
# by default. Execute the following command to start it,
# and to cause it to launch at startup:
#
# sudo launchctl load -w  /Library/LaunchDaemons/org.macports.postgresql82-server.plist 
###########################################################  
--->  Installing postgresql82-server 8.2.6_0

To create a database instance, after install do
 sudo mkdir -p /opt/local/var/db/postgresql82/defaultdb
 sudo chown postgres:postgres  /opt/local/var/db/postgresql82/defaultdb 
 sudo su postgres -c '/opt/local/lib/postgresql82/bin/initdb -D  /opt/local/var/db/postgresql82/defaultdb'

And here’s my test to determine if su was working:

 sudo su postgres -c id

I got no output… So I tried

 sudo -u postgres id

And I get:

 uid=401(postgres) gid=401(postgres) groups=401(postgres)

So now what?

There’s an easy way to deal with this: Grab the server admin tools from Apple and install them.

Next, fire up Workgroup Manager (its in your /Applications/Server folder). Ignore the dialog and go straight to the menu and select Server -> View Directories. Select the PostgreSQL Server user in the left-hand pane, then select the Advanced tab and change the Login Shell to something like /bin/sh. See the picture for details. Don’t forget to hit Save.

Now the command from before works:

# sudo su postgres -c id
uid=401(postgres) gid=401(postgres) groups=401(postgres)

So now we can finish the last step to initialize the database, then fire up Postgres.

  sudo su postgres -c '/opt/local/lib/postgresql82/bin/initdb -D /opt/local/var/db/postgresql82/defaultdb'

If you already ran the launchctl command, you can make Postgres start with these two commands:

 sudo launchctl unload -w /Library/LaunchDaemons/org.macports.postgresql82-server.plist
 sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql82-server.plist

Otherwise just run the load version of the command.