Picasa Download Script

First of all, this is a mess. It uses a bunch of junk in 'daveperl', which is like my personal collection of perl idioms. Stuff like cat_file() that I use all the time and hate re-coding. I need to extricate this script from that stuff before it can really be useful to other people.

It used to use some Google Picasa perl library, but I grew really impatient with it. It's coded up in some really weird ways that do things like dynamically create perl objects and functions out of the XML that gets spit back from the Picasa API. It is pretty arcane stuff that I never did get working right mostly because it is a deal project.

The current stuff (I think) depends on the CPAN Picasa library and not the Google stuff any longer. But, there may be a dependency on the Google stuff in a place or two.

http://sr71.net/projects/picasa/

It basically goes and finds all the Picasa pictures, then sees if it has local copies. If not, it downloads them and sticks them in a directory. It caches things like the album modification time so it can speed things up not having to download the contents of *each* album during every execution. It uses filenames and exif timestamps to determine uniqueness. I don't do checksums because those change with picture rotation. If anyone finds this compelling or useful, let me know. I'll start trying to clean it up a bit so others might get some more use out of it.

Freestanding Server

People have asked a few times if there is a freestanding server or some other program to which the Eye-Fi card can upload. No, not really.

Here's how the Eye-Fi card and service work, at least in the way that I have them set up. This differs for the Eye-Fi "Home" card, or if you don't have the "upload to Internet" (or whatever it is really called) enabled.

1. You take a picture, and dump it on your card
2. Your card notices the picture gets on the network and looks for your PC. If it can't find your PC, it uploads directly to api.eye.fi (an Eye-Fi managed server on the internet)
3. If it can find your PC running the Eye-Fi Manager, it uploads to your PC and lets the PC upload to Eye-Fi. The PC keeps a copy if you asked it to.
   
4. The Eye-Fi server looks at your preferences and uploads pictures to Picasa, Flickr or whatever you want.
   

"Gallery" which I think is a collection of CGI script to manager photo uploads is an option as a destination to which Eye-Fi can upload. So, you can have Eye-Fi upload to a server that you run, but your pictures pass through their servers on the way. I haven't bothered doing this. When I first went and looked at Gallery, it had some big scary security holes in it that scared me away.

There were a few people working on a server to replace the Eye-Fi Manager. I have not heard about that effort in a while. There are some security mechisms built into the card to keep it from uploading to unauthorized software which make this hard to do. I don't think this is Eye-Fi being nasty or hating Linux or anything. It's a genuine security feature to keep the "bad guys" from tricking your card into sending all its pictures to the evil hackers. I would love to write a open replacement for this part of the Eye-Fi Manager, but I'm not sure how feasible that would be and still ensure that the "bad guys" don't do the same. I'd be more than willing to give it a shot if I had some idea how the card authentication worked.

What I do personally is that I trust Eye-Fi with my pictures. I let them go to Eye-Fi, then get uploaded to Picasa. I paid the $20/year or whatever it was to get 10GB of extra storage at Google. Then, I use the official Picasa API and some perl to download the pictures from Picasa back to my web server where I stick them in my own photo album. The downside to this is that I have to trust Google and Eye-Fi with my pictures along the way. The upside is that when I take a picture, it automatically gets backed up in three different places.

Eye-Fi Config Release 006

Someone asked for this, so here you go!

This basically just adds WEP support. It only handles the WEP-128 keys specified as hex. This was not hard to add since the EyeFi card has most of the smarts to do the network type selection internally.

http://sr71.net/projects/eyefi/eyefi-config-006.tar.gz

I still need to figure out how to get the firmware updated without going into Windows. There' some experimental code in here, but it does not work.

Is the "Anniversary Edition" really SD or SDHC?

My "big" camera is a Nikon D50. It is a wonderful camera, but it is getting a bit dated and only takes regular SD cards, not SDHC. As I understand it, the SD specification is limited to 2GB, but the hardware can handle a full 4GB of addressing. The SD standards people artificially lowered the standard to 2GB either to move people to the SDHC standard faster or for some technical reason I don't yet understand. Are there technical issues with some devices and 4GB FAT16 partitions?

There is also evidently a requirement in the standard that SDHC cards be FAT32-formatted. Older cameras (like the D50) can't read this filesystem even if it were present on a small capacity card. The neat part of all of this is that some "4GB SDHC" cards are actually plain old SD cards electrically and just rebadged as SDHC since a 4GB card isn't allowed to have the 'SD' logo. What I'm saying here is all speculation on my part that I've pieced together from some very reliable web forums. :)

There have been some reports that, if you get one of these cards, simply formatting a 4GB FAT32 SDHC card as the older FAT16 will make it work in cameras like my D50. I really wonder if this new Anniversary Edition card is one of these special beasts. It would make sense for Eye-Fi to do this since "all" they'd have to do is replace the on board flash with a bigger chip and wouldn't have to change the card electrically to support true SDHC. For $79 shipped at Costco, I just may have to find out!

Latest Progress

I have not been doing very much Eye-Fi hacking lately. I do with I had been able to activate my new card without going into Windows, but that was really a one-time inconvenience. Not nearly as bad as some devices that need constant plugging into Windows. BTW, there's nothing especially wrong with Windows, I just don't like rebooting.

I had a Belkin N-1 Vision that turned out to be a pretty expensive ($179 when I bought it) piece of junk. My first one locked up (bad hardware?), and my second one seems to be unable to do IPSec passthrough. So, I just dropped in the Actiontec MI424-WR that came with my Verizon FiOS service. When the Eye-Fi card tries to upload through it, both devices reset themselves!

In all fairness, the Actiontec seems much more fragile than the Eye-Fi, and I do know that the Eye-Fi intentionally reboots sometimes to clean up transient error conditions. That's fine behavior for the card since it reboots fast and you can still take pictures. It is much less nice for a router to do that.

I also stumbled upon a neat blog entry about the benefits of open source. When I read it, I immediately thought of my little Eye-Fi card. I truly hope some day our friends at Eye-Fi decide to seize upon the community of really clever people out there that would love to build upon their fantastic devices.

Oh, and when is the Eye-Fi iPhone app going to be released?

Geotagging Works!

I took a picture earlier this morning, which wasn't geotagged. I just took another and noticed that my AP had finally gotten into Skyhook's DB. I submitted it last Tuesday and it took a bit less than a week.

If you wonder what this is, I've been making cables to convert my old RAZR chargers to charge the iPhone 3G. The RAZR is a male USB Mini-B, so I need female USB Mini-B connectors which have been really hard to come by. I've been using this guy's schematic to get the right voltages on the USB data pins so the iPhone will actually charge.

P.S. Please no stalking! :)

Zaurus Development

Someone loaned me a Sharp Zaurus SL-5600 so that I might try a port of my software to it. I got an arm cross-compiler, but I can't even get "Hello World" running. It just segfaults. I bet I'm compiling for the wrong platform or something. Oh, well. I really need to get a NIC for this thing. Should make development a lot easier.

I also picked up an iPhone 3G, which got me thinking. Seeing as at least a couple of the Eye-Fi employees have iPhones and that there's already a Mac OS port of the Eye-Fi manager software, I have to wonder if we'll be seeing an Eye-Fi iPhone app some time. The biggest sticking point is that neither the iPhone or the Eye-Fi card can act as a WiFi access point as far as I know, they can only join them.

I also haven't been able to get Loki to work. As far as I can tell, there's no Linux port. Sigh... I did submit my AP to their DB, though.

Eye-Fi Explore and release 005

My old classic Eye-Fi card broke after I squished it, plugged it about 30,000 times and cut a hunk of plastic out of it. The guys at Eye-Fi replaced this "defective" (their words, not mine :) and to show they are swell folks, replaced it with one of the newfangled Eye-Fi Explore cards that do geotagging. Thanks guys! BTW, I don't consider it defective, I consider it abused.

I was able to do all of the card initialization in WINE. It would not complete the final test step, but it works fine anyway. I need to dig through those WINE logs and see if there's anything interesting in them.

New observations:

1. I have two access points running with the same ESSID, but on different channels. This works for my PCs, but made the card very mad. It would not associate for long enough to upload anything.
   
2. The card comes from the factory without its EyeFi/{reqm,rspm,...} files. The manager software must create them. I created them with dd, and was able to use the card before I ran the manager software. I also saw a 'EyeFiManufacturing' wireless network in the card log!
3. I added a network to the card before I registered it. The card DHCP'd and tried to upload its pictures, but choked once it talked to the Eye-Fi server. No surprise here, but it does open up the possibilities if someone ever reverse-engineers the server protocol.
4. I enabled geotagging, but it does not seem to work for me, yet. Anybody have this working? Anybody know of any quick ways to check Skyhooks's AP list?
   

I'm also pleased to announce Eye-Fi Configuration Tool Version 005. This should fix the crashing bugs that some people were seeing. My printf() function was screwed up and calling printf() instead of vprintf(). Git tree here.

Git tree

Because someone asked, I've also put up a git tree. I'm honestly not planning on putting out that many updates, but I figure I'll share anyway.

http://git.sr71.net/cgi-bin/gitweb.cgi?p=eyefi-config.git;a=summary

Eye-Fi Config Release 004

There are no new features or anything in here, but I wanted to at least share my recent work. I put the file that I'm using for CHDK in here, and also some bits intended to help get this running on OSX. Thanks got to Reed Wade for the work he's doing on OSX!

http://sr71.net/projects/eyefi/eyefi-config-004.tar.gz

This is completely untested because my Eye-Fi card is now completely dead. I'll probably go get one of the new "Explore" cards at some point, but I'm without a card for now.

A kind soul also lent me a Sharp Zaurus to try and get my software running on it. I'll let everyone know how that goes.

Shaky, but working again

I got the card running again with the volume serial number 'AA52-6922'. I guess that's how the Eye-Fi manager actually finds the card, rather than looking for the EyeFi/r* files like my software has been. Anybody know offhand how to find this stuff in Linux?

I also got a request to make my manager software work on OSX. I'd love to do that, but I don't have a Mac. If anyone wants to give me ssh access to one or something, I'd be happy to give it a shot. Shouldn't be more than an hour or two of work, as long as there are development tools around that I can use.

Even the Eye-Fi Manager Can't Find the Card

I think all of my formatting and messing with the card made it unrecognizable from the Eye-Fi Manager's point of view:

06/14/2008 13:28:35 000000B4 INFO Scanning initial drives...
06/14/2008 13:28:35 000000B4 INFO Not Eye-Fi Volume serial number
06/14/2008 13:28:35 000000B4 INFO Not Eye-Fi Volume serial number
06/14/2008 13:28:35 000000B4 INFO Drive scan is complete...

Could someone who has the card running in windows do a 'dir' of the Eye-Fi card for me? I think I wiped out its serial number and need to recreate it.

c:\> dir e:
Volume in drive E...
Volume Serial Number is XXXX-XXXX
...

Could you send me the X's?

Busted Card?

Well, I think I've finally worn out my Eye-Fi card. It is throwing all kinds of I/O errors and getting corrupted filesystems pretty regularly. It won't even read in the camera sometimes. This was happening even before I carved the hunk out of it, yesterday. Oh, well... maybe I'll try one of the new ones when they come out. I think I've abused this one enough. :)

Working CHDK Code

Well, it didn't take long, but I have CHDK running accessing the Eye-Fi Card. For now, I just had it dump the firmware version string, but the sky's the limit! I should have it scanning and adding networks in no time.

As a commenter noted, CHDK requires that the lock switch be flipped before it can be run off an SD card. The Eye-Fi has no such switch. So, I took a chunk out of mine with an X-Acto Knife. Here are some pictures. I've also taken to sticking clumps of electrical tape over my new hole if I need to stick the card back in a non-Canon camera.

I also stuck some foil into the Eye-Fi reader to fake the lock tab because, with that chunk out, all the readers think the card is read-only. I could have done that with any reader, but my "official" Eye-Fi one was already taken apart. I wonder if I can override that in software. Anybody know offhand?

It took hacking my Eye-Fi Config program up quite a bit to keep it from using some of the facilities like abort() that are not available in CHDK. But, most of the code essentially stayed the same, and I integrated it pretty well. Now I just need to get some menus coded up to navigate the various options.

Now that I have bits of aluminum foil, electrical tape, razor blades ant toothpicks (for digging bits of tape out of SD card slots) I really feel like MacGyver.

CHDK Port

Anybody out there ever use CHDK? It is an open replacement firmware for Canon cameras. I'm getting a Canon SD870IS, and there is evidently a port to that model. I think I'll have to port my Eye-Fi Config software over to it and see if I can get it to work. That should allow me to add access points out in the field.

Status File

I think I just figured out how the Eye-Fi and the D60 talk to each other. There's a one byte file on the card, next to the RSPM, RSPC... files called simply 'status'. You need to create it, but once it is there, the card will fill things into it.

When no pictures are pending for upload, it has a 0 in it. I've also seen it at 4 and 5. My guess is that 4 means "trying to upload" and 5 means "failed to upload, not trying any more". I'll have to explore more to be sure.

The D60 can read this file and keep the card powered up as long as the card is trying to upload. As soon as the upload finished, a 0 appears in the file, and the D60 knows to power it down.

More on Power Consumption

I attempted to splice my ammeter directly between my D50's battery and the camera with some electrical tape and aluminum foil (seriously). It worked, but the ammeter dropped ~0.6v out of 7.4v that the battery puts out, and that was enough to keep the camera from being able to take pictures, but it could operate in standby.

In any case, the camera itself draws about as much current as the EyeFi card does. I think the real power sucking this. The camera (either off or in "standby") with the "Meter-Off" takes a very small amount of power. Powered on, it takes probably 40 or 50 times more. Following those EyeFi directions means that you put your camera into this higher power consumption mode for 30 minutes if you take one picture instead of 16 seconds like it might have before.

So, my guess is that the power consumption comes more from the camera itself than from the EyeFi card. But, the only reason to put your camera in that mode is for the card in the first place. Anyway, I bet this is exactly why EyeFi and Nikon are working together on the D60 and EyeFi card integration.

Eye-Fi Power Consumption

I got a very small, 6 inch USB extension cable from my favorite cable store. Then, I chopped it up to get access to the power supply wire and clamp my cheapo ammeter on to it. I plugged a card reader into my PC with this contraption. Note that I'm measuring the USB power supply at ~5v, not the 3.3v that the card actually gets and that these are very unscientific measurements made with Radio Shack equipment.

My card reader alone: ~31 mA (unchanged if a normal SD card is simply plugged in)
After plugging Eye-Fi card in (no pending uploads): 56 mA (25 mA for the card)

All the numbers that I'll state from here on include only the amount that the Eye-Fi card is using after subtracting the amount for the reader.

Then, I took a picture to make the start start an upload and turn on its radio. The ammeter was bouncing all over the place, but it peaked at over 170 mA and I'd guess that the card averaged about 120 mA during the upload. The strange part is that, after the upload, the power usage only goes back to 40 mA. It never (at least in a couple of minutes) gets back to the 25 mA where it started. I wonder if the radio never gets completely turned off. Hmm.

My Nikon D50 has a 7.4v 1500 mAh battery. Let's be stupid and pessimistic and assume that the card is running at its very peak power of ~170 mA and round that 5v USB voltage up to 7.4v. That's still almost 9 hours of theoretical power. I can't possibly be seeing that effect on my battery life, right?

Is my methodology screwy? Any electrical engineers out there that want to tell me how much of an idiot I am? Email me.

New Eye-Fi Products

I was offline last week, attending a few family members' college graduations. While I was gone, Eye-Fi announced a few new products. Just wild conjecture, but I bet that the only difference in the cards is the firmware. I doubt we are seeing a hardware rev at all. In any case, looks like Eye-Fi has some good marketing folks! I love to see companies that put out solid incremental products like this.

The Eye-Fi Explore can also now log on to certain WiFi hotspots but apparently no T-Mobile or AT&T ones like at Starbucks. This is great progress and I hope to see these capabilities widening in the future, but this is definitely a baby step. More wild conjecture, but I bet that Wayport either already has or was able to implement a simplified log-on which the Eye-Fi card could perform. Remember, the card has very little RAM and a pretty tiny program storage area. It has no real web browser and can't do the SSL that most web sites use to secure authentication information.

The "Explore" can also evidently geotag your photos if the WiFi access point you are using has been mapped. According to this page, the tagging is done in the "Eye-Fi Service", which I assume means it happens after the upload from the card. I bet there's an extra bit of data sent along with the access point's MAC address during the upload now. I'd love to get some dumps of these uploads happening if anyone has one!

Eye-Fi on Vacation

I just returned from a lovely vacation in Playa del Carmen, Mexico. The hotel I stayed at had Wifi, and I took my trusty D50, Eye-Fi card, and my Linux laptop. But, I never actually used the card. I stuck with my trusty non-Wifi Sandisk card.

Why? Well, I promised my lovely wife that the laptop would stay in its bag except for checking flights (we were affected by American's MD-80 issues). It's a bit hard to program the new network names into the card without the laptop, so it went unused. I also did not want the card sitting in the camera without using its wireless capabilities because the only time I've ever been concerned with battery life in my SLR has been with the Eye-Fi card in it. Contrary to what is in the FAQ I do seem to feel the battery life shorten significantly with the card in there. I wish I had an A/C adapter and an ammeter for my camera to measure this.

Anyway, I really do wish there was some way to configure the card out in the field, or even shut off the radio. I'd love to not have to lug my laptop around (I really need an iPhone). Perhaps the card could populate the image folders on the card with partial ESSIDs for the open wireless networks that it sees. If I changed the active folder on the camera over to one of those, the card could start using the new network. I bet this would be nearly impossible to get working on all cameras, but it might work on a good subset.

Maybe other features of the card could be enabled or disabled by having the user view and delete pictures on the card. Could the card detect when the camera deletes a picture that had an image of "RADIO OFF" in it and actually turn off the radio?

That's really why I'd love to hack the firmware. I could customize the card for me and my own perverted personal use like I do with my kernel.

Eye-Fi Config Version 003

New Features since 002:

• use standalone crypto implementation for PKCS#5 from wpa_passphrase. Keeps us from having to depend on running wpa_passphrase with system()
• better error reporting when mount points can not be found
• fixed an error with eyefi_mount[] getting set when there is no real card there
• moved card initialization order around so that it can actually use debugging from -d flag

Download here

Booting the Eye-Fi Firmware

Has anyone out there ever run eCos or Redboot under a qemu instance? I got a version of little-endian Malta Redboot, and got it to run. But, I think the Eye-Fi card is big-endian.

What I'd like to do is get the Eye-Fi firmware booting in qemu so that I can start trying to modify it. If figure that if I can get another Redboot to load the Eye-Fi firmware image and run it, them I'll halfway there.

I just had someone email me and ask if it was possible to use the Eye-Fi card in a remote location where there is no Internet access, and just upload to the PC. Sadly, I don't think it is quite possible with the current firmware. One of the first things the card does when it boots is check with the api.eye.fi server to see if the card has been registered. Without that step, I don't think it functions. Update! I'm wrong! :)

OK, I appear to have been completely wrong. Jason Justman pointed out that he's done just this. I tried to reproduce what he has done by blocking my Eye-Fi card from getting to the Internet on my router. It still managed to upload to the Eye-Fi manager on my PC (which was not blocked). I'd love to know if anyone can reproduce this or has any other tips.

Eye-Fi Config Version 002

Infinitely more features!! Changes from 001:

• Automatically discover mount point (works in Linux, definitely needs portability)
  
• Actually list the log command-line option in the help

Download here

Tool Available for Download

I've uploaded an early version of my configuration tool so that others can play with it. It's available from my web site.

Please download any play with it and do not forget to report bugs back to me. At this stage, I'm sure there are plenty of them. I'm also happy to take patches if I'm missing features or you find behavior annoying.

Chris Davies' software is much more mature than mine, so I'll probably switch over to using his once he releases it. But, you can all at least play with my version for now.

Really sneaky card snooping

I just did something I thought was pretty neat. I was trying to validate that my approach of dumping the card log was OK. This is mostly because I get some gunk out of there sometimes, and it just doesn't look like completely correct data, like I'm missing something. It is hard to do an apples-to-apples comparison with the official software logs because the log on the card is constantly being written, and you can never fetch the exact same data twice.

So, I turned off all the file writing in my control application. Then, I started it up like normal and told it to do a log dump. It sat in its loop, waiting to get the control sequence that it never wrote back from the card. (Remember that the manager writes a number to a file, then expects to get that same number back when the command is done). It'll never get out of that loop, right? Well, no.

Simultaneously, I told the manager to dump the card's log. My software sat there and read the files along with the manager, but I let the manager do all the writing. My software walked along in lock step with the manager as it went about its business dumping the entire log. This let my software do the reading, but the official software do the writing (and its own reading).

What I ended up with was two virtually identical logs, except for the last byte, which my software truncated. I need to go fix that now.

Handling corrupt JPEGs

If you haven't it figured out, yet, I'm lazy. But, I've been taking my card out of the PC reader and putting it in my camera to take a picture every time I want to get the card to go onto the network. I decided to just create a JPEG on the card instead of actually taking pictures. The quickest way to do that was just to dd some random gunk into a JPEG like this:

    dd if=/dev/urandom of=/media/EYE-FI/dcim/100dchef/dsc_0000.jpg

That works, technically. The card even tries to upload it. But, the server gets a bit annoyed because, somehow, I guess random data isn't a valid JPEG very often. :)

[00:54] Starting to upload "/public/DCIM/100DCHEF/DSC_0000.JPG"...
[00:54] Server has 0 of 77312 bytes for DSC_0000.JPG.tar (file ID 779166).
[00:55] Uploaded 77312 bytes for DSC_0000.JPG.tar. Waiting for the response...
[00:55] Probing for listener at 10.6.0.60:59278.
[00:56] Returning from upload function with error 28.
[00:56] Starting to upload "/public/DCIM/100DCHEF/DSC_0000.JPG"...
...

It will apparently sit in that loop for quite a while. It did until I stopped it by deleting the picture. I bet the card does not quite have the resources to validate JPEGs itself, so it just uploads the raw files to the server. The server returns with that "error 28" and the persistent little card retries.

It might be nice to have a limit on these retries. But, I don't think it's quite that simple. You want the card to be persistent. Wireless isn't exactly a lossless transmission medium, and its bound to have problems uploading from time to time. It's a matter of figuring out when to retry and how much. It might be reasonable, though, to have some special handling of corrupt files. TCP should keep the files consistent during transmission, and I would imagine that most of the corrupt files that the server sees are also corrupt files on the card. There isn't much reason to keep trying to upload these.

Card Registration in WINE

I realized the other day that I'd left out a critical portion of the whole process: card registration. You still need the EyeFi-supplied software to register the card for the first time, or to add and remove online services. It might be nice in the future if EyeFi could make registering or adding and removing photo services a web-only procedure that didn't require running the manager software at all. I'm pretty sure the card doesn't actually get consulted when you change the web services, so this should be possible in theory.

I deleted my account, and recreated it. Interestingly, the software didn't to anything to the card when the account was deleted. That seems to indicate to me that all of the smarts about the card registration are on the EyeFi API server. The manager software doesn't do anything like tell the card when it is not registered any more. The card can figure that out on its own from the server.

This is supported by the fact that the very first thing the card does after it gets power or sees a new picture is do a DNS query for api.eye.fi, then POSTs some XML to it with its MAC address and a field called "cnonce" that looks like an MD5. These folks have looked into how this works in more detail.

I wouldn't mind getting my hands on another brand-new card to see exactly what the card's behavior is when it is fresh out of the box. I wasn't smart enough to trace it from the beginning.

Interesting EyeFi Articles

Here are a couple of articles about the EyeFi card from a security researcher's point of view. They're very interesting and do make a few good points about how the card functions and what a few of its limitations are. It definitely taught me a few things about how the card works.

They also make the point that the EyeFi developers "get it" when it comes to security. That's very high praise, and I'm very glad to hear that they've been so responsive.

Thanks for the links, Ben!

Mad card

During my "add network" testing, I screwed up the command a couple of times by putting the network key in the wrong place. This effectively put random gunk in the "key length" field and caused all sorts of fun. The card got a bit warm and eventually, I got this in the logs:

[08:51] IEEE 802.1X RX: version=1 type=3 length=95
[08:51]   EAPOL-Key type=254
[08:51] WPA: RX message 1 of 4-Way Handshake from 00:11:95:18:... (ver=1)
[08:51] WPA: Sending EAPOL-Key 2/4
[08:53] Disconnected from WLAN (reason = 4).
[08:53] Done scanning for new photos. Uploads not pending.
[08:53]   EAPOL-Key type=254
[08:54] Done scanning for all photos. Uploads not pending.
[08:54] Connection dropped (previous state is 2)
[08:54] Connecting to WLAN "bcdefghijkl"...
[00:00] Eye-Fi firmware 1.0741 Feb 15 2008 17:13:41 started, hardware revision 1, 2016768/32768 KB storage.
[00:00] Card was reset due to an exception or assertion failure at PC = 0x800029f4. Dump saved.
[00:03] 11216 bytes allocated, 34524 bytes free (34524 bytes largest)
[00:03] Card is in online and desktop transfer mode (timestamp 1203796494).

I wonder if it needs a bit more sanity checking when it is parsing network add data. Although, I can't really fault it for having trouble when I poke basically random data into it. It's really cool that it does recover so nicely.

In any case, I do have add network working quite reliably now.

Dump Log Command

If your EyeFi card is malfunctioning and you contact support, they'll likely ask you to dump the card's log. You can do this from the tray icon in Windows.

The first step in dumping a log is issuing an 'o' command. The 'o' command is the EyeFi's own ioctl(). It is a multiplexed command that can do several things depending on the byte after the 'o'. 0x1 seems to fetch the card's MAC address, for instance. An "o 0x7" command is always issued right before the log is dumped. It always (as far as I have seen) returns 0x00010000. That happens to be the length of the log: ~64k. But, the RSPM file is only 16k in length. So, how do we dump a 64k log 16k at a time?

'm' is the log dump command. It looks like this:

struct dump_log {
char cmd; // 'm'
u32 offset;
};

The card's MIPS cpu runs in big-endian mode (MIPS is bi-endian), so that offset needs to be byte-swapped if you're running the control software on an x86 machine.

The result in RSPM is all ASCII, except for the first 8 bytes for offset 0; I thought these were garbage for a while, but when I started comparing the logs to those obtained from Windows I realized that my logs were shifted around. The end of my logs didn't contain the latest entry; it was somewhere in the middle. It turns out that those first 8 bytes are actually two 4-byte numbers giving two offsets into the log data. The log data returned from this command are actually a circular buffer and those 8 bytes specify its start and end. The log file actually begins and ends in the middle of the buffer. So, the data that comes back from the card after issuing 4 of those 'm' commands looks like this:

struct log_buf {
u32 log_end_offset;
u32 log_start_offset;
char data[16384*4-8];
};

There was one last thing. The logs I got back were a bit smaller than the Windows logs. That's because, coming back from the card, the data has UNIX-style carriage returns. But, the manager turns them into DOS-style before writing the file.

Add Network Working

As I said, I've been playing with Chris Davies' software a bit. I just got the 'a' command working for the first time, which is the add network command. It's a lot more simple than I thought a first. It's a real advantage decoding this stuff when you have so much smart logic on the card.

Most of what we first thought were complex encodings for adding the card appear to really just be garbage left in memory from previous operations. Even watching the Windows software after a log is dumped, you sometimes see the log data come back in the results of subsequent commands. This makes things a bit harder to decode and it would be really nice if the card zeroed things out, first.

Anyway, in the case of add network, the command is a struct like this:

struct add_net {
char cmd; // always 'a'
char net_name_len;
char net_name[32]; // always null padded
char net_password_len;
char net_password_hex[32];
};

Notice that there isn't a field for the encryption type, channel, MAC address of the AP or anything. The card evidently figures this all out internally, which is really nice.

Oh, and you know how you actually have to let the manager software successfully test for a network before you can add it? I think that's a bit of a disadvantage sometimes. Like if you're going over to your Aunt Betty's house and you know her essid before you go, you can't program it into the card in advance. You would need to be there before you can add it. This is a pretty user-friendly feature because it ensures that people don't mistakenly add networks and get confused about why it never works when they just entered the password wrong. I think generally this is way nicer than what I have on my desktop today.

Well, that feature looks to actually be implemented in the manager software, not the card. I believe you can just issue an 'add' command to the card without actually testing it, first. The manager software just happens to always test it in advance. It might be nice to have a "force-add" feature in the future.

The lights go on

OK, I've been stabbing in the dark on the OS all day, but I think I have it now. No, really. Really.
Check out this Atheros Bulletin. Atheros provides eCos for the ar6001 chips and they call it "eCOS BSP - Basic Kernel." There are references to the "BSP version AR6000-..." in the firmware *and* eCos has a 'devfs'. Then, there's this post from Berend about using lwIP with eCos.

Any eCos hackers out there?

Firmware Fetching

Thanks to EyeFi's Berend and his kind comments explaining the difference between "pure zlib data" and gzip formats I've put together a script to download from the EyeFi web site and decode a copy of the firmware using a random card's mac address. Now if I can just figure out what format the firmware is in...

I also made a mistake. Reading wpa_supplicant's license agreement, I saw the GPL in there and missed the part about it being dual licensed. However, the nice folks at EyeFi were quick to point out:

The Eye-Fi software utilizes LGPL, BSD, Apache, MIT, ZLib, IJG (Independent JPEG Group) and Boost licensed open-source components. None of these licenses require disclosure of the source code.

So, I guess there's no Linux in there after all.

They also pointed out that EyeFi is acting in good faith and sharing improvements they make to open-source components that they use. This bug filing would seem to indicate that they use IwIP for their TCP/IP stack. This might be why p0f doesn't seem to be able to detect the card's OS.

Firmware, press releases and the GPL

First of all, it's great to know that some of the EyeFi engineers are reading this! Berend (who I think is one of the EyeFi Founders) pointed out that:

The API call simply returns the data in gzipped and Base64-encoded format.

That's what I suspected, but I can't seem to get uudecode to give me anything that looks like gzip in its output. The sizes line up, but there's no gzip header.

Berend also noted that the bits about the chipset being Altheros and MIPS are well documented in press releases. Cool! I'm not just making this stuff up! Let's quote from there:

The solutions also come with full driver support for Windows Mobile CE 6.0 and Linux-based designs.

WinCE does not have a 'devfs' as far as I know, so that would lead me to believe that the card is either running Linux or EyeFi has ported another OS to it. That would be a wee bit interesting if it ran Linux.

Although, the card can't be running Linux since it is GPL'd and would require EyeFi to publish the source code for at least the kernel. They aren't doing that as far as I know, so it can't be running Linux. Just in case, here's a nice informative link which clearly explains a company's obligations under the GPL.

(changed "a commenter" to refer to Berend)

Firmware image

I was able to sniff the latest update to the firmware as it happened. It does download it as far as I can tell from the site I mentioned in an earlier post. However, I can't quite figure out what the download format is.

But, I did manage to catch the manager writing 'EYEFIFWU.BIN' to the root of the card, just before issuing a 'b' command to reboot. I believe this let me get a hold of a complete, decoded firmware image. There's a lot of interesting stuff in there. The card's chipset looks to be a Atheros ar6000 which contains a MIPS core.

It also appears that the card has /public and /private mount points that the card's OS can see. /public is what we get to see on the card itself, and private is where the real fun is stored. There also appears to be a 'devfs'. devfs's exist on at least FreeBSD and older Linux versions, and this doesn't really point in either direction.

But, I don't really see an OS, a bootloader or any executables in the image. I ran 'file' at 1-byte offsets on it and didn't really come up with much. I think it could certainly use another pair of eyes.

Working Standalone Software!!

So, a reader of this blog has been able to understand many of the interactions with the card. He's written software to:

1. List configured networks
2. List scanned networks
3. Print card MAC address
4. Print card firmware information

and I've added on to it to fetch the card logs. This is really good progress, and the ability to add networks to the card shouldn't be too horribly far away. I'm sure that the author will be willing to share it publicly soon. I'd post it somewhere, but I don't want to be passing around someone else's code for which I have no permission to distribute. I'll post here as soon as it is available.

BTW, a recent commenter noted that the card can only add ASCII WEP keys, and not hex ones. They didin't get much help from EyeFi support. I've seen the same limitation, and I'm happy to say that the card's hardware accepts hex keys. It's just a limitation of the manager software right now that it only takes ASCII. I've actually watched the manager software program the hex values derived from the ASCII keys into the card.

Firmware updates

There have been some people talking in the comments about things to do to the firmware. While I'm sure some of you have actually agreed to the license and are likely restrained by its terms, I bet most of you haven't. You also don't necessarily have to agree to a license to simply download files from a web site.

Simply sitting in an airport next to me (or stalking me at my house) you could easily pick up my EyeFi card, its mac address, and watch it do its firmware updates. You might also see the manager fetch a file from the server for its firmware updates in plain text. If you want a firmware file for a souvenir, you could try fetching from these urls which are completely unauthenticated (except that your card has to be registered). So, try a few random ones until you get a hit:

for ((i=0;i<100;i++));
   url="http://api.eye.fi/api/rest/eyeserver/v1";
   rmac=$(printf '00-18-56-03-%02x-%02x\n'
          $((RANDOM%256)) $((RANDOM%256)));
   wget "${url}/getCardFirmware?Card=$rmac&Version=1.0496";
done

The error result documents are ~100 bytes, while the real ones are a couple hundred kilobytes.

Why I care about EyeFi on Linux

A recent commenter noted that EyeFi simply supports XP and OSX because of the volume on those platforms. It would be silly for them to support the minuscule number of Linux users. Also, most Linux users at least have one XP machine on which to set up the card.

First of all, I kinda hope EyeFi does not release Linux software. We are a small minority and they surely will waste engineering resources on an operating system that they do not understand. I do hope they release information on how the card works. That will let me set it up myself, or let hundreds of others do the same.

Sure, I could go steal my wife's machine to set up the card. But, what do I do when I'm traveling? Isn't that the time I need to change settings in the card the most?

Card communication on the local network

Chris Davies saw this blog and was nice enough to point me to some tools he has written. They somewhat take the place of the Eye-Fi manager. In Chris's words:

> It's all fairly trivial. After the card gets a DHCP
> response, it sends out a flood of ARP requests ... for
> each host in the subnet.  It then does a triple handshake
> on the predesignated port of each host in turn, until it
> finds one that's running a server on it. After that it
> just tries to authenticate with that host, and send it
> pictures.

"That host" is, of course, the one running the Eye-Fi manager. Chris's tools will listen on that predesignated port for the card to contact it, and start accepting pictures.

Well, here's the problem: I don't see any of this behavior!! I set up a second ethernet card in one of my Linux machines and turned on bridging with it and a normal card, then put my wifi access point on that second card. That makes my PC act like an ethernet switch that allows me to monitor all the traffic that goes over it with network monitoring tools.

I used a filter in Wireshark to narrow down the traffic to just the Eye-Fi card with this filter

(eth.addr == 00:18:56:aa:bb:cc) &&
!(ip.addr == 216.218.219.2)

That's because "00:18:56:aa:bb:cc" is the MAC address of the Eye-Fi card, and "216.218.219.2" is the IP address of the Eye-Fi server. So, this let me look at all of the card's communication with things other than the Eye-Fi server.

After the DNS requests to talk to the Eye-Fi server, I don't see the card do much of anything.

Wine patch

Well, it turns out that wine doesn't respect the Windows CreateFile() flags FILE_FLAG_NO_BUFFERING or FILE_FLAG_WRITE_THROUGH. I have a really simple wine patch to "fix" that. I really haven't thought it through in too much detail, but it seems to work.

Basically, whenever wine sees either of those two flags, it just disables all of Linux's filesystem caching with the open() flag O_DIRECT. You can apply the patch to the latest version of wine out of its git repository. and probably earlier versions, too. Here's my patch.

I'll probably be trying to get this into the upstream wine distribution soon.

BTW, if you want some help building wine, I'd be happy to help, so just email me.

Eye-Fi Manager Working in Linux!!!

First of all, this is perhaps the most significant progress I've made so far in unraveling this device. Getting it to work in Linux means that I have a very complete set of debugging tools to use on it to see how it works.

I'm running the "Eye-Fi Manager.exe" with wine. The issues that I was seeing with it were because Linux is not reading back the actual disk contents, but cached disk contents. I noticed this by enabling debugging on the usb_storage driver, then accessing the card multiple times with both O_SYNC and O_DIRECT set.

Those options should have caused Linux to do real reads from the card each time, but I didn't see the usb_storage debugging kicking in each time I did a read. See this post if you're curious why it's important to do real reads each time.

Anyway, I instructed wine to have the kernel throw away its cached copies just before each read() of the control files. I did it with this call:

posix_fadvise(unix_handle, 0, 0, POSIX_FADV_DONTNEED);

The result: a working Eye-Fi Manager in Linux. You need to hack and rebuild wine, but it is at least a proof of concept. It works!!

Now, I just need to go find out why the reads() aren't obeying O_DIRECT. It's probably either the usb_storage driver or the vfat filesystem code.

Alternative to Upload to Computer

Since I run Linux don't have a Windows machine, I can't simply use the Eye-Fi manager's ability to copy all pictures down to a PC. But, I noticed that Google's Picasa has an open API for accessing it (and 1GB of free storage). So, I changed my picture service to Picasa and started digging. There is an apparently Google-written implementation of a perl libarary to access Picasa.

I used this library to create a little script called picasa-upload.pl. You can run this script to automatically download all pictures directly from picasa, and store them on your PC. You can get my script along with a slightly hacked up version of the Picasa libary here:

http://www.sr71.net/~dave/projects/picasa-download

Run this every few minutes in a cronjob and, for me, it's just as good as the built-in Eye-Fi capability!

Eyefi software uses Curl!

I was grepping through the Windows binaries looking to see from where messages come when I ran across a little surprise. The Eye-Fi manager software uses a piece of free software called curl. Specifically, they statically link in a library portion of curl, libcurl. It basically implements a little tiny web browser inside of their application so it can talk back to the Eye-Fi web servers.

It is a bit of a shame that Eye-Fi would choose to use free software inside of its products, but neglect Free Software users by not providing specs or software that run on Free operating systems.

By the way, nobody is violating any license or legal terms here. Curl is free to use in the way Eye-Fi uses it. I just don't think it is very nice.

Request encoding

I've noticed that the data written to the REQM file changes dramatically, despite the command being the exact same. I can only assume that there's some kind of encoding going on there. Maybe not encryption per-se, but there something scrambling it, and with something more than just the simple command. There's some value that changes each time getting thrown into the calculation. But, without going in and doing something against the license agreement like disassembling, I can't think of any easy way to figure out what is going on in there. Suggestions welcome!