|
DMI News |
Previous Entry.. Next Entry.. Server Reload and other Fun Stuff
May 15, 2014 17:42 Over the last several months I've had some mysterious server crashes. It took several crashes and monitoring of logs to realize that the HD was having issues, and some more events to realize that it was dying. I decided to start doing more frequent backups and had started the process of poking the hosting provider toward getting that replaced, but since reloading the system is always a timeconsuming, tedious, and annoying activity, I hadn't gotten in a huge hurry about it. Then one day reddit links to my site and things go absolutely nuts. I did some tweaking to the server to handle the load, and all things being considered, everything handled quite nicely. And in the midst of all of this, the HD decided this was the PERFECT time to just crash. No, really, reddit did NOT crash the website, it was just really bad timing, but I'm sure that's how it looked from the other end. Annoying, to say the least. In my own defense, from the time of crash, through a complete fresh installation on a new server, reuploading all of the backups to the new drive, reconfiguring the new system, and getting everything configured, I mostly had everything up and running again in less than 4 hours, although some things took me a few more days. Both through the crash, and the sudden influx of traffic and online media attention, these events have spurned more site activity and encouraged me toward revitalizing the site again. Thus a new adventure begins. First, some of the quirks. This is MOSTLY just poor planning on my part, but setting up the web server for this site is never an easy task. Although I've made an effort in recent years to reprogram the backend, there is still much that is at least somewhat reliant on 15 year old hack jobs that were required to make things work back in the beginning, when I was on a completely different network configuration, on a vastly different distribution, with older libraries, far less traffic, and practically no security concerns. First quirk is always a permissions issue. There's always a program or a database file that I keep stored in some strange place, like 3 layers deep in my home directory, while everything else is stored in the root web folder or an extra directory specified for that purpose. I'm constantly making an effort to move away from those little issues when I find them, but about the only time they pop up is during a reload, and they're so easily fixed with a change in the directory permissions, that it's really easy to once again just forget about them. Secondly, libraries. I use a LOT of them, and none of them come preinstalled on a stock CENTOS install, so I spend a couple hours running yum everytime I get a runtime error. Often it's easier to just recompile everything as well, as the header files are often missing too, and the compiler usually gives more useful error messages to what I'm missing. Remember all of those -devel respositories as well. YOU NEED THEM. I've been avoiding 64 bit systems for some time now, mostly because I know my source code is going to have porting issues, and there's not much need to push in that direction as long as I'm using less than 4 gigs of ram, but some other providers aren't even offering 32 bit installations anymore, so I figured I had better start getting prepped for it, just in case. I have a 64 bit version of CENTOS installed on one of my home servers for this reason, and I downloaded the source and gave a shot at compiling it. It compiled just fine, but runnign it... not so much. Thankfully, most of my code is pretty conversion friendly. The long type is now 64 bits instead of 32, but int is still 32 bits, and I use that type the most anyway, so there won't be any issues there. The one that took me a day to figure out is a consequence of laziness on my part, and one I'm going to have to rectify. When making a function call, the C compiler will look for that function defined, either implicitly in the same file, as the function itself, or explicitly through a single line that defines the types for the parameters and return variables. If neither is available, C just guesses, and most of the time, those guesses are integers. The good news is, that on a 32 bit system, almost every variable type is 32 bits. Int, long, pointers, floats, are all 32 bit variables, so casting them to integers and back to whatever they're supposed to be isn't a problem. It's bad programming, but it still works. As it happens, and what caught me, much of my code contains functions that return pointers to data, and as I mentioned, I'm lazy, and have not explicitly defined my functions of my utility files, so those functions that returned pointers were getting those pointers truncated as it tried to cast the 64 bit pointer to a 32 bit integer and back to a 64 bit pointer. So now I'm going to have to go through and type out the definitions for about 500 functions, probably only 30% of which I actually use anymore. This will probably be a grand opportunity to eliminate some cruft as well. I'm pushing forward on some of the projects again. I had the Critter running for a few hours last Friday, and realized why I haven't had it operational for a while. So now I'm working on the authentication system so I can more easily control who has access to it and prevent abuse. I ordered a new video server so I could get the living room camera functional again, only to find it doesn't work, so I'll be sending it back, and possibly trying another one. I'm going to poke my ISP again and see what options are available for more upstream bandwidth. It's been a few years now. Will probably also check to see if DSL options have improved at all in this area. They keep sending me the propoganda saying that it has... I know better, but who knows.
|