unable to add or delete files on TS-Linux embedded single board computer

Hi everyone,

My company has been using ARM boards loaded with TS-Linux for some time now (http://www.embeddedarm.com/products/...oduct=TS-7250#), and we have had very few problems. It's a bare-bones OS so there's not much that can go wrong (relatively speaking).

Today, however, I was trying to upload a file to a board (using sftp) and it failed. Using ssh to log into the board as root, I saw the the flash memory was 100% full -- thus the problem. Things got weird when I tried to delete a file to make room.

So, here's a summary of what I tried and the results:

$ df -h
Filesystem   Size  Used Available Use% Mounted on
/dev/root   29.0M 29.0M         0 100% /
$ ls
BPU.exe   RunBPU.log   Savefile.xml   bpu.xml
$ rm BPU.exe
rm: unable to remove `BPU.exe': Directory not empty

(This is the first hint that something strange is up - BPU.exe is clearly not a directory. But fine, if it wants to be treated like a directory, I can play ball...)

$ rm -r BPU.exe
rm: unable to remove `BPU.exe': Directory not empty
$ ls
RunBPU.log   Savefile.xml   bpu.xml
$ df
Filesystem   Size  Used Available Use% Mounted on
/dev/root   29.0M 29.0M         0 100% /
$ ls BPU.exe
BPU.exe
$ ls -a
.   ..   RubBPU.log   Savefile.xml   bpu.xml

So basically, the file throws an error that doesn't make any sense when I try to remove it. It then disappears from the ls search, unless I point to it directly, in which case it is still there. It is also still taking up space on the disk. The same thing happens for every file I have tried. What's more, when I reboot the system and log back in, all the files are back as if I had never touched them.

I haven't the faintest clue what is going on here. I have tried rebooting the system, changing the permissions on the files, and even unplugging the thing for a few minutes. No luck. They are like ghost files. I can make them invisible, but I can't kill them, and they always come back.

Any ideas would be greatly appreciated.

Thanks!

Nevermind. I was under the wrong assumption.

I'm not entirely sure. I have to admit I am a physicist by trade, and am learning all this software stuff as I go.

What are the different modes, and how do I check that?

I will add, in case it helps, that the board I am using does not have a hard disk, in the classic sense. It has 32MB SDRAM and 32MB flash memory, where the kernel and file structure are stored.

Nevermind. I was under the wrong assumption.

This is an ARM based single board computer that runs an embedded version of linux. It is not tinycore and AFAIK TC runs only on x86 based hardware.

I have several TS-7200 boards but have not run into this problem yet. It is possible that the on board flash memory has gone bad and would suggest calling Technologic Systems to ask for assistance. I assume this problem is unique to a particular board. Have other boards produced similar behavior?

Originally Posted by michaelk This is an ARM based single board computer that runs an embedded version of linux. It is not tinycore and AFAIK TC runs only on x86 based hardware. I have several TS-7200 boards but have not run into this problem yet. It is possible that the on board flash memory has gone bad and would suggest calling Technologic Systems to ask for assistance. I assume this problem is unique to a particular board. Have other boards produced similar behavior?

You are right. My apologies.

I do seem to recall reading about this problem somewhere, and I think it goes something like this: The journalling filesystem has to write something to the disk related to the journal, before it can do the erasure, but since the disk is full, it fails and cannot do the erasure. I can only imagine that I read this in the TS-7000 mailing list on Yahoo groups. You might be able to find something in the archives there.

Can you unmount the drive, and rebuild the filesystem with fdisk and mkfs?

--- rod.

Thanks for the input!

Yes, michaelk, my first thought was also that the on board flash memory has gone bad, but I was hoping I was mistaken. We have almost 30 of these boards which have been running for over a year, and this is the first time I have seen this issue.

Your idea also seems plausible, theNbomr, so I will try rebuilding the file system as you suggest. My one hesitation: due to a lack of robust error-checking in our upload scripts, the boards are overloaded to 100% of their disk capacity semi-regularly. It is usually not an issue, I just have to clear some space and they work again. Since this one is behaving differently, I am inclined to believe it is a hardware failure rather than a journaling issue. Still, it couldn't hurt to try.

Thanks for your help, I'll report any new findings as they come.

JD

Update: I did a little digging in the TS-7000 mailing list archives that theNbomr mentioned, and found a very helpful thread: http://tech.groups.yahoo.com/group/ts-7000/message/2349 (you may or may not have to join the yahoo group to view the thread). It turns out this is a problem with the YAFFS file system. The dmesg log shows the following error (don't know why I didn't check this earlier):

!!!!!!!!! Allocator out !!!!!!!!!!!!!!!!!
yaffs tragedy: no space during gc

Apparently the problem can be fixed by saving the file contents elsewhere and using flash_eraseall to reset the flash partition. However, this does not address a related issue, namely that YAFFS marks bad blocks far too conservatively, eliminating blocks that are actually ok and reducing the amount of space available over time. The eraseall utility will not touch blocks that are marked bad, so they can not be recovered.

For the brave at heart: Eddie Dawydiuk hacked the kernel and eraseall utility and posted modified versions so that you can recover those bad sectors (ftp://oz.embeddedarm.com/tmp/erase_BB/). Beware though, as this method might also recover blocks that really are bad, which could be, well, bad.