|
DMI News |
Previous Entry.. Next Entry.. CRASH!
February 07, 2003 01:19 What started as a simple hardware upgrade turned into a mess. I wanted to install a 100mbps card instead of the 10mbps I had in there. The good news is, the 100mbps card is now installed. The bad news is.... *sigh* Booting back up, it loaded the wrong kernel. no problem, I'll just have to reboot again, so I wait until it finishes booting. Well, it got stuck in an infinite loop running a couple scripts. I'm not sure why exactly, but it was probably something I did wrong. I can't log in to stop the runaway process, nor can I do the three finger salute to do a clean shutdown, so I hit the power switch. Coming back up the second time, this time choosing the right kernel from lilo. Turns out that one was wrong too, so I figure it'll get stuck in the same trap. I wait for it to finish booting anyway, just in case I have better luck this time. It gets to init, and gets stuck there. Peculiar. I boot off a floppy to get in there to see if the inittab file or something is acting screwy. Comment out most of the stuff in rc.local just for good measure and try to reboot again. No dice. Still getting stuck at inittab. Well, its possible to mess up my root directory. It happens. And it wouldn't be the first time I've reloaded the OS, so I figure its easist to just back up the few system config files and reload it, a 2 hour process at most. It had been a few years anyway, so why not. So I mount up my partitions to find a convienent place to copy stuff too. And it would turn out the most likely candidate is my 88 gig paritition, /dev/hdc4 which has PLENTY of free space. And that's where I ran into trouble. mount gives me TONS of errors. That's the first sinking feeling. Then it finally gets mounted and I do a directory listing of the root directory. Well, it sorta looked like a directory listing. Maybe it would have looked better while standing on my head. Root directory was nuked. Inode 2 has been fubared. My life just got a lot more difficult. I fire up debugfs to see if I can make some sense out of the mess. Even if the root directory is messed up, I can still go in there and rebuild it if I can find the inodes for the directories linked under it, and this isn't TOO hard to do, because all you have to do is find ANY directory on the drive and follow the toplevel inode link up. How I wish it was that easy. Unfortunately, it would appear that fsck, while booting up, decided that since there wasn't anything linked from the top directory, that all the inodes were really free space, and proceeded to clear all the inodes. And I mean ALL of them. Now I'm feeling sick. The problem here, is the two most important directories on the system were on that drive. All the webspace and my home directory, which contains, among other things, all the source code for all my projects. The semi good news is that I had a recent backup of the website, and SOME of the code was copied in there, and other source is availble elsewhere. In fact, most of the source I'm currently lacking is a bunch of small cgi applications that I can reproduce if I need to. 400 lines of code max. However, I may not have to. Although the inode table on the drive is effectively worthless, the data itself is all still there. I can't directly access it anymore, but I CAN access the partition device and read the entire drive byte by byte, including all the source. So I have dd scanning the drive and grepping for #include lines, which typically will mark the top of my source files. Since most of my source files are less than 4k in size, they'll fit within the datablock and I won't have to search for a bunch of pieces. Of course, all the media I had stored on that drive is gone, but that's only a minor annoyance. Seeing how I can't even remember what any of it is, I'm not planning on losing any sleep over it. Now, here's the lesson for everyone, including myself. BACKUP! no.. really. BACK YOUR SHIT UP! This is a really shitty way of losing a lot of work, and a totally unnecessary one. I got lucky. It could have been worse. But I'm going to make every possible effort to be sure it never is again. But at least the network card is working! :)
|