Apr 5, 2014

How I solved my SD card corruption problems by modding the Raspberry Pi

I've spent last month struggling (once again, after some trouble with the power) with SD card corruptions due to the Raspberry Pi's "design idiosyncrasy" (a funny name to avoid cursing).
Until I finally found a solution... 




My setup was an embedded system based on Raspbian OS. It was supposed to be light (carried on a backpack), autonomous and battery powered. It would automatically run a custom C++ (openFrameworks) application on start-up, work for half an hour playing sounds in reaction to a joystick's axes/button changes, and then be safely shut down. And so on, several times a day.

LONG STORY
I included a PiTFT just for monitoring purposes (with the Pi booting in terminal mode, no desktop), to tell the user when the system was ready, and also when it was safe to remove the power plug. The PiTFT wiki tells also how to install a module that allows to safely shut down raspbian with a pushbutton connected to GPIO pin 23 and ground.




That works also without the PiTFT.



I must say PiTFT was a bad choice. The communication with the TFT driver slows down the system a lot, so I couldn't refresh the screen and simultaneously play 8 sounds without getting nasty glitches on the audio. Instead, not needing the touch overlay, I should have chosen a cheap 3.5" TFT monitor plugged to the TV out connector (like explained in Adnan's blog), but that's another story...
My main concern was that, sometimes, the shutdown button didn't trigger the shutdown process, so the user (with basic computer knowledge) was stuck unless he runned for a keyboard, hit CTRL-C to kill my app, and then run a shutdown command in the terminal. If he, instead, unplugged the power, the SD card got corrupted, and I had to restore the backed up image onto the card again (no point to tell him to do that).

That isn't something you can justify at all. You can't tell the user: "well, you know, things are not perfect, this is a known issue in many systems using SD as the main OS storage, blah, blah,...".
The only reasonable speech would be: "I'm sorry I f*ck*d it up by choosing this f*ck*ng computer instead of using a regular laptop thus adding another kilo to your shoulders (besides the overkill in computing power). Maybe I should have learnt to programm the app in an android device, rooting it and making it communicate with the joystick, with the overall increase in developing time and expenses".

I followed some tutorials and instructions to avoid Raspbian to write on the card, with no success, an example could be:
I didn't try to move the system to an USB drive. I could also have used the Industrial Perennial Environment, but the updating/debugging process would have been really painful.

Mounting the file system as read only (read this, this and this) would also have been a pain in the ass.
Ideally, the SD card write protection switch would be the perfect solution, but it is not a hardware lock: it is finally the system who decides whether to respect the switch and write or not... and it does write... because the RPi design is not using that SD holder's pins at all!!!.
So I gave up and finally substituted the  [RPi + joystick] with [Arduino + Wave Shield + Pushbuttons + pots], and reduced the system functionality to play just one sound each time. End of the story for that system/setup.
A few days later, my subconsciousness (which was still working on it) came up (too late... bad boy!) with a new possible solution: why not adding a hardware switch to disable the SD card just before unplugging the power?... 

I made a short research about SD interface pinouts and functionality and learned that there is a CS (Chip Select) pin that tells the card whether to listen (CS=LOW) or not (CS=HIGH) to the commands/data on the other pins. I also read that almost all SD cards have an internal pull-up resistor on that pin, so, if the pin is disconnected from the host (floating in the air), it would automatically go to the "unselected" state, blocking any read/write action and preserving the integrity of the card.
"Well" - I said - "here is the chance to get rid of all the troubles this little bastard has caused"...

And here comes the final solution to my Raspberry Pi SD card corruption issues:


WARNING: this is a dirty hack involving a hardware mod of the board. Do not do it. If you try to do it, it will be at your own risk and responsibility. I don't care if you end up with a useless board/card/whatever. This mod is intended for setups where the OS/apps don't need to read/write the SD card. If you setup needs to write data (temp files, logs, etc.) you can try using an additional storage media, such as a pen drive (I haven't tested that).

On the RPi schematics, the SD card holder has the CS (Chip Select) pin (named DAT 3/CD) connected to GPIO 53 through resistor R46.



I unsoldered that pin of the holder and put a switch in between.




RESULTS:

The board/OS behaviours on different scenarios are:

- Plugging power with switch OFF (CS disabled, that is HIGH due to the SD card internal pull-up resistor), the board acts as if the SD card was not present: no problem. If I, with the power still plugged, turn the swich on, the board does nothing: it already stopped the boot process when not finding any boot files.

- Plugging power with switch ON (CS enabled by the board, that is LOW through resistor R46), the board boots normally as expected and the OS is able to read/write the SD card.

- ONCE THE OS HAS FINISHED BOOTING, turning the switch OFF, resulted in the OS not being able to read/write the SD card. But, as it was set up to have all temp and log files in memory, I had no problem so far. The test I did was to open a terminal and make a simple copy of a file, with the CS switch OFF:

The copying process stuck but the system remained healthy. Hit CTRL-C killing the process, and there was no copy of the file. Then turned the switch back on and repeated the command: no problem at all, file copied, system/card didn't complain.

- FINAL TEST: let the system boot normally with the CS switch ON and the app start, turn the CS switch OFF and then unplug the power without shutting down the app/OS. Result: no SD corruption, system healthy and rock solid, happy, happy. 
F*CK YOU, LITTLE BOARD, I WIN!

CONCLUSIONS:

I've spend precious time of my life fighting with the Raspberry Pi (of course, let us be positive, I learned a lot in the process). I'm still waiting for a small cheap linux-ready tablet with rooted USB ports for my next embedded projects. In he meanwhile I'm also playing with an Olinuxino Lime board (Allwinner A10 CPU), which is supposed to be able to boot/use a SATA drive as main OS storage. Next versions of this board are also planned to come with the OS in a NAND flash memory (I'm currently trying to add the memory chip myself like in this thread). I may also buy a Cubieboard2, with an A20 CPU and 4Gb NAND flash for the OS.


FURTHER WORK/RESEARCH:

There is still a lot to test with this mod and I hope it also works for other people/project and save their time.

It might also work (but that I haven't tried):
- Modding the SD card holder's SD_CARD_DET pins instead of the CS
- Set GPIO 53 to HIGH with the application to block the SD card access by software.
- Connect the unused COM2/WP pins (which detect the card's write protection switch) to some other GPIO and, at boot time, mount the file system in read only or read/write accordingly.
- Design a circuit that holds enough energy to set CS pin to 3.3volt in the lapse when the voltage drops below say 4.5volt, right after unplugging the power, but before the system tries to write its shit to the card.

Happy hacking, have fun!!!

6 comments:

  1. Did you try mount the filesystem as readonly?

    ReplyDelete
    Replies
    1. There are many side effects on mounting the filesystem as read-only (you have to prepare the system for that, avoiding all the system's writes).

      Delete
  2. Would it be possible to have a large capacitor on the power supply, such that on a power cut out this preserved power for several seconds, during which time CS could be switched off, say by a comparator (eg 741 op amp)?

    ReplyDelete
  3. Would it be possible to use a large capacitor to back up the power supply, such that it persisted for several seconds after power was cut. During this time a comparator could sense this, and switch off CS?

    ReplyDelete
    Replies
    1. Actually, I tried that without success. Probably because the capacitor I tested wasn't big enough to make a proper discharge curve to let the circuit latch the CS line.
      Is the thing written in the "further work" notes as:
      "Design a circuit that holds enough energy to set CS pin to 3.3volt in the lapse when the voltage drops below say 4.5volt, right after unplugging the power, but before the system tries to write its shit to the card."

      Delete