|
DMI News |
Previous Entry.. Next Entry.. And so it Begins.... Sorta
June 16, 2009 02:20 After a LONG year of coding, feet draging, and endless hesitation, I have finally taken the plunge. I have started implementing the new application servers to handle various features on the site. And this means what to you? Absolutely NOTHING. First off, I've only implemented one server so far, out of 12 that I have planned, and I've only implemented features that were already available before. Yes. I spent a whole year just to get to this point. And I'm not even done yet. Err.. what's the big deal again? If you're not interested in programming or network management or similar topics... um.. just... go away. Still here? Ok, here's the deal. Last June my server went down for 6 days due to a explosion at the datacenter where it's located. While I was able to get it fully functional again, the downtime had a signficant negative effect on the exposure of the site. By the time I got it restored, I had lost 2/3 of my regular traffic. So what I started doing was to implement a system by which I could easily and quickly set up a secondary server. Since then, I have been paying for two servers, one of which does practically nothing, but it's there, and it holds frequent backups of the main server, so should something drastic happen again, I can hopefully have the site back up and operational in a matter of hours, instead of days. But it got me to thinking, why shouldn't I have a realtime updated server, ready to fire up and go at a moment's notice, with completely up to date data. And that brought me to the next phase in that evolution, why not have both servers operatinging simultaniously, with live updated databases, and able to handle twice the traffic I currently do. Why not indeed. The inspiration then lead to a great deal of perspiration. See, most of the site has been very poorly designed. Yes, I did it. I'm at fault here. It's not that I WANTED to design it poorly, it's that I never quite realized it was going to get as big as it ended up, and the more dynamic I made it, the more difficult it became to loadbalance over multiple servers. While I have active database programs for some of the realtime data, much of the data is accessed directly from files. I did try some database packages in the past... however, that idea fell apart in testing. It basically boiled down to the fact that no matter what I tried to do, I was going to have to rewrite the entire system. So, if I was going to do that anyway, I was going to make sure I only did it once, and when I did it, it would be robust enough to last forever and handle a virtually unlimited number of concurrent visitors. For instance, the image server I'm currently putting together will be able to handle 1 trillion images. That doesn't mean I expect to even reach a tiny fraction of that number, and it's currently several hundred times larger than the number of images that photobucket and other sites currently handle, and they've been specifically designed to handle it. I just don't want to get caught again, stuck in a rut because while I could easily improve the site's traffic, there's no way it could currently handle it. I can't handle any media attention and reap the financial benefits that might result from such exposure again. So, I'm making sure, this time, once the system is complete, I'll be able to grow it easily and seamlessly by doing nothing more than adding more servers to the mix. The second issue is one of efficiency. There are many places on the site where I'm literally doing a linear search on thousands of records searching for a record, and I do this several times everytime a page is loaded. Yes, it's stupid. Yes, it's high school level datastructures. It's not that I don't know these things. But it was a quick and easy implementation when I had only 20 records to sift through. I honestly never expected to need more at the time. As previously mentioned, I won't be making a similar mistake. In every instance, I'm assuming a worst case scenario, and preparing for it, so the software can handle it in the most efficient way possible, within physical limits. Needless to say, and I've already said it, it requires me to rewrite virtually the entire system. All 138,000 lines of code. Yikes. So here's the good news. Let me introduce DMICMS. The DMI Cluster Management System. DMICMS offers a framework to allow the quick and easy developement server applications with an automated system in place to connect to a master DMICMS server and other servers on other networks, and keep all of the data up to date on all notes of the cluster. It also allows for more servers to be added at any time simply by connecting to the network and starting up the associated application. And, worst case, if the master node dies, a secondary node will automatically take over and continue operation with minimal downtime (a minute or less). The idea here, is that this functionality is implemented completely via library calls, and the actual functional parts of the server don't have to concern with it. I've also developed a new format for passing commands and data between the servers and also between the servers and clients. I've kept it as simple as possible, not only to keep the coding for it to a minimum, but so anyone wanting to write a compatible application won't have to go through a great deal of trouble to do so. Each command is in the format of a single header line which contains data in a http encoded string, like a URL query string, and a raw imagebuffer of unlimited size. Each connection can be separate or can use keepalive to maintain an open connection and pass multiple commands on a single connection. It might seem obvious to programmers of network applications, but I had not developed a library to handle all of these actions and pretty much had to reinvent it from scratch everytime I started coding a new server application, or I just kept the networking aspect of things as simple and useless as possible. For this reason, much of my databases remain in flat files simply because the time I would have to spend to implement each of them into a server application seemed to not be worth the trouble, no matter what benefits I might achieve from it. However, all of that is in the past. I've got my first functional server working, and although I'm still making some tweaks and getting it fully functional before I move onto the next one, I'm happy with the progress and the speed at which I was able to crank it out. I'm hopeful I will be able to crank out a new server application every week or two, although some of them will doubtless take longer (lampmaster is going to be a nightmare). Still, I have hopes that I can get them finished by mid-fall, so I can concentrate on plans for a Xmas display this year. I've been wanting to do one for years now, but the site has never been up to the challenge. Maybe this year it will be.
|