DonationCoder 2018 Infrastructure Remodel

March 25, 2018 - by Jesse Reichler

---

In January of 2018, DonationCoder.com completed a major overhaul of our website infrastructure. This involved 1) Moving physical servers and hosting companies; 2) Moving thirteen years of static web pages to a proper content management system; and 3) Completely restructuring a secondary server that we use to provide web accounts for coders associated with the site.

This essay documents that process, the decisions that were made along the way, and some lessons learned.

A brief history of DonationCoder.com server hosting

When DonationCoder.com initially went online in 2005, it was created using static hand-coded html web pages, and hosted on a cheap $20/month shared web hosting plan somewhere I can’t remember. At the time, the website consisted of only a handful of web pages and a discussion forum. The hand-coded static html pages had the advantage of being fast and requiring few server resources, and the small number of simple pages made the site easy to update via ftp. In 2005 this wasn’t such a crazy way of creating a website.

2006

Sometime in 2006 we moved to LiquidWeb hosting, using a virtual private server (VPS). A virtual private server is essentially a virtual machine where your server is isolated by software for security but shares physical hardware with other clients managed by the hosting company. While affordable, performance problems can be very difficult to track down because you don’t have access to the physical hardware and you have no control over, or insight into, the resources used by other clients on the machine.

2010

In 2010 we were ready to move to a more powerful and reliable server. At this point we moved to a dedicated server, where we were the only user running on the physical server. We chose SoftLayer, because we were focused on eliminating the mystery slowdown issues we occasionally faced on our virtual private server, and SoftLayer had an excellent reputation for reliability and uptime, and for a high quality network.

In 2010, at the time of the move to SoftLayer, we had a DC member, Gothic, who was a skilled linux administrator, and he enthusiastically set up everything for us. We were on an unmanaged plan at SoftLayer, and Gothic was our volunteer server administrator. We eventually went with a fairly non-standard setup -- we installed VMWARE ESX (a virtual machine hosting solution), and ran several virtual servers of our own on our dedicated physical machine. This let us set up the main donationcoder server on one virtual machine, isolated from a handful of additional virtual servers we set up for hosting other community websites and projects. It was at this point that we began creating web accounts for other members of our community to host their software.

We were liberal about giving out web space and email accounts on the new server and over time we had several dozen accounts on our member server. We basically let long-time members of our site have web space and an FTP account and let them upload whatever they wanted.

2012

As early as 2012 it was clear to me that we were on an unsustainable path regarding our server infrastructure and management process.

The biggest problem was that our volunteer DC server admin was simply not available to maintain and administer the server, except in emergency situations. While everything worked, we had no one managing the server, no one doing regular software/security updates, etc. For all of the wonderful strengths of linux, it’s still not at a point where the free linux distributions can keep themselves updated without an experienced server admin at the helm. Our situation was made somewhat worse by the fact that while our servers were optimized well for our needs, they had been set up idiosyncratically and in non-standard ways. Where we previously benefited from being able to do everything we wanted the way we wanted, I could see the writing on the wall that this was all coming back to haunt us going forward. Everything was still working just fine, but we were already at the point where we were afraid to upgrade things or change anything on the server for fear it would lead to a cascade of breakages that we wouldn’t be able to fix. Most of all, I was growing increasingly concerned that we were going to be hit by a security issue that needed prompt attention without someone to handle it.

Hiring a system administrator or trying to find another volunteer was an option, and we probably should have done that as an interim step, but at the time it seemed to me that given the custom and idiosyncratic nature of our server setup, we would be better off starting from scratch on a new server. I dreaded the day that we might have a catastrophic failure on the server -- such as a hardware failure -- because while we did have daily backups of forum content and the static web pages, we were not at all prepared to recreate the server itself. So not only were we unprepared to fix problems on our current server, we were also completely unprepared to move to a new server where problems could be fixed in a timely fashion. Worse yet, should a disaster occur that would prevent us from having access to our current running server, moving would be orders of magnitude more difficult and would have to be done under severe time pressure. The feeling of being “trapped” on our current server, and helpless if things went wrong, was palpable and haunted me.

In addition, it was becoming clear to me that our anything-goes doling out of member server web (and email) accounts on our secondary (virtual) server was out of control. For one thing, these dozens of accounts were all set up in an ad-hoc manner. There was no central record of all of them -- some had email accounts, some had FTP accounts, some had shell login accounts. Some had domain names of their own, some had mysql databases, etc. Furthermore, we had no policy about what software could be installed by members, how much space could be used, etc. It is a credit to the group of people that call DonationCoder home that nothing was ever abused, but it was a completely unsustainable situation, and the security risks were real. Checking on the member accounts, it was clear that many of them were using out-of-date installations of software that had major security updates pending. There were tons of abandoned and broken web pages. And we had no good way to see what was happening on the member server, so when it occasionally slogged to a crawl we had no idea what was causing it.

There was another serious problem with the way we were doing things, unrelated to the server administration and setup, and it had to do with the way the content on our main website was organized. While the static web pages were not out of place on the internet in 2005, by now we were acutely aware of their limitations, and the growing number of pages on our site (several hundred at this point) was exacerbating the problem. The site navigation and menu structure was hand coded and made adding new pages or sections painful -- so painful that I began to avoid it at all costs. And while my old school process of uploading new page versions via FTP worked fine when I was the only person editing pages -- it made it near impossible to allow others to update pages on our website. While these were not issues when the website was started and was maintained entirely by me, over time these limitations began to feel like a straightjacket, limiting what we could do and making it impossible to let others help manage content.

The basic path forward was pretty clear as far back as 2012:

• We needed to move to a new clean server (or servers), starting from scratch and following a more standardized process so we could set everything up in a clean, orderly, maintainable way.
• We needed to move to a managed provider, so we had someone we could call upon to keep the server updated and secure.
• We needed to do major house cleaning on all the member accounts we had created, standardize them and set up policies that would ensure that all software was up to date and secure and not abusing server resources, and most of all could be maintained long-term.
• We needed to recreate all of the main site content and web pages using a proper modern content management system (CMS), which would allow me (and others) to maintain the site more easily. We needed to clean up and organize all the kludges and custom helper scripts that had been implemented over the years.
• We needed to establish a more consistent, simple process for maintaining the server that would let us migrate between hosting companies more easily in the future.

2012-2014

Sometime in the 2012-2014 period I began looking seriously at content management systems (CMS), and took a few aborted first steps towards recreating our web content in a CMS. I had some experience with Drupal and I flirted with using it, but ultimately decided that my frustrations with Drupal were too large to overcome. I played with a dozen different CMS systems but never found one that I loved.

My frustration with existing CMS systems reached a peak in 2013 and resulted in perhaps the biggest derailment of the modernization process. Like many coders, I suffer from the curse of having a low tolerance for other people’s code, and I decided that I should build a new CMS framework designed to help coders create community user-driven websites. I began such a project, open sourced in Python, called Mewlo, and spent a solid year on it before deciding that it was simply not realistic for me to maintain and finish such a project, and that it was not a practical solution for modernized DonationCoder.

The sheer number of web pages that needed conversion was overwhelming and in the end it was hard to get motivated to move the content when the existing content was still working -- however fragile it felt. Moving existing content and doing server administration was the last thing I wanted to work on -- I was much more interested in programming applications -- and DonationCoder does not have the money to hire someone to do the work.

Nevertheless, it was becoming increasingly clear that we didn’t have an infinite amount of time to move on our own terms. Sooner or later something was going to happen to require us to move servers -- the only question was whether we could get it done relatively painlessly on our own timetable, or whether it would be under duress after a major server crash or security breach.

2015

After letting go of the dream of coding a custom CMS framework, I returned to the search for the perfect CMS. My general experience evaluating CMS systems has been one of gritting my teeth and trying not to get too frustrated by design decisions that rub me the wrong way. I suffer from the problem that many coders suffer from -- a constant nagging feeling that I cannot live with other people’s design decisions that are different from the way I would do it. I considered other languages (Python primarily, since after C++ it’s the language I use most often, but also Java, Go, and others), but in the end, despite all of its weaknesses, I returned again to PHP based on its straightforward deployment with web servers, and the fact that we were using a PHP-based forum system (SimpleMachines) and had lots of existing custom PHP code related to the website.

It has been said that when you choose who to marry you are choosing which particular set of flaws you can live with. I feel that way about frameworks. In evaluating software I have always followed the adage of focusing not on the things that are possible to do, but rather on the things that the software makes easy/intuitive to do. From a programmer’s perspective, most modern CMS systems are flexible enough that with sufficient effort you can get them to do most things. For me the two primary questions were, how easy and intuitive was it to do the small handful of things that were relevant for the DonationCoder server, and how easy and straightforward would it be for me to write and maintain custom extensions to do the more unusual stuff that might be unique to our needs. At its heart, leaving aside the forum, the DonationCoder website is just a large hierarchy of simple web pages -- most of which describe an individual application available for download. The most common thing we need to do is add new pages, edit existing pages, and reorganize pages into a clean hierarchy.

Choosing Joomla

After trying several CMS frameworks that I rapidly gave up on, I finally settled on using Joomla sometime in 2016. I had looked at Joomla once many years ago during a messy open source fork from the Mambo project. My expectations were low, and I remembered Joomla and Mambo as inferior products. But in using it on a test server, I was impressed by how intuitive the back-end administration section felt and at how much more mature the project was. The core functionality -- management of menus, articles, and pages -- felt intuitive and logical. The programming API seemed reasonably well developed and logical. A nice breadcrumb system made it easy for people to see where they were on the site. The built-in search functions are imperfect but sufficient. These may seem like basic features but Joomla was a good match for the structure of our content. Development of Joomla is active and responsive to security issues and the updating process is painless.

Another key issue that one faces when it comes to frameworks is the quality and quantity of third-party extensions. The community of extension/plugin developers for Joomla is fairly active, and for advanced features that I was determined not to have to write -- like a gallery system -- reasonably mature and actively supported third-party extensions existed. And just as importantly, it turned out to be fairly painless for me to write custom extensions of various sorts for any special needs we might have. In fact I open sourced several of these extensions in January 2018. To be sure, there is no perfect CMS, and I could easily have continued my search indefinitely, but the time had come to pick the best of the worst and simply make forward progress. I left open the possibility that we would eventually get to the point where Joomla would prove itself unsuitable, but I’m happy to say that in the end it proved to be well suited to our particular needs.

Not that there aren’t things I would like to see improved in Joomla, such as the search functionality. Mostly I’d say that the extension system could use some substantial improvement in order to simplify the process of creating, packaging, and installing extensions (I’ve been impressed at how clean and straightforward the WordPress extension process is).

Migrating content

One key to finally breaking breaking past the resistance to moving the content was DonationCoder forum member Stephen66515, whose enthusiasm for the migration project exceeded my own and who was not burdened by my fear of committing to a particular content management system. Stephen volunteered to lead the process of recreating content and started converting web pages on his own. The sheer number of pages was a bit overwhelming but the two of us eventually made enough progress that we could at least imagine the day when it would all be converted.

Confident that we had found an acceptable CMS, I began to convert and rewrite some of the custom web scripts that I had written over the years (donation payment processing and license key generation), as plugins for our Joomla CMS system.

2017

Our work migrating website page content and coding custom extensions proceeded slowly -- exasperatingly so -- in fits and starts in our spare time and was punctuated by long periods of becoming unmotivated and doing nothing but second-guessing the effort. Finally by 2017 we had made enough progress that I could at least imagine finishing, and I was able to work full-time on the migration project. 2017 became the year where I was really able to focus my energy on getting everything moved over and doing that final 20% of the work that takes 80% of the time. By October of 2017 we had essentially recreated all of the content of the site on a temporary test server in our new, heavily customized, Joomla-based CMS.

Other than final testing and tweaking, this took care of the issue of moving website content to a new CMS. However, in addition to converting all of the main website content from static pages to a CMS system, we also wanted to move physical servers and reorganize our member server accounts.

Moving Servers

There were several questions that had to be answered related to moving servers and hosting companies:

• What hardware did we need?
• How much could we afford to pay each month?
• What hosting company should we use?
• How should we split up the main site from the member accounts?
• What kind of management services and control panel did we need?

Some of these questions had easy answers and did not require too much thought. Having experienced the existential fear of not having a server administrator, it was clear we needed to go with a managed solution, where we could call upon tech support to keep our server updated with the latest security patches, etc., and to handle emergencies. I also wanted several layers of automated daily backups for redundancy. In addition, I wanted to have some control over the server without having to do everything via the command line. So this meant a managed server with a proper control panel. After trying a few I settled on cPanel for the control panel. This added about $20/month per server to the hosting costs, but was the right decision for us. While there are several good paid and free control panel options available, this was an area outside of my comfort zone and I felt most comfortable with the user interface and level of control provided by cPanel; it lets me see what’s going on, and manage the parts of the server I feel comfortable managing.

As for choosing a hosting company, I dragged my feet and agonized for months. The world of web server hosting can be overwhelming. Every company tells you they have the best service and performance, but it is near impossible to know what is really going on underneath the hood. While you can compare the hardware specs of particular servers, what you can’t see is how good the network infrastructure is, how oversold the bandwidth is, how good the support folks are, etc.

With an abundance of money, you can guarantee yourself a solid hosting company. This is one of those cases where money will buy you quality if money is no object. But DonationCoder is not in such a position, and it was important for us to maximize our value on the dollar. Luckily we do receive enough money from donations that we can afford good hosting and don’t have to settle for some of the really questionable hosting companies.

My first pick was LiquidWeb, which we had used as our hosting company way back in 2006. While we had some performance issues with them at that time, that was due primarily to being on a shared (VPS) server, whereas now we would be on our own dedicated server. I received a good deal on a 3-month lease of a LiquidWeb server to do a test setup, and began some negotiation with LiquidWeb over a long term hosting contract. My experience with LiquidWeb was positive - they have good people and and a well run organization that feels polished and professional. The people I talked with were always straightforward and I never got the unpleasant car salesman treatment that I would often encounter from other hosting companies, with misleading price quotes and terms.

The only problem with LiquidWeb was that it was on the upper end of the cost spectrum for us. So I continued to look for a trustworthy, reliable hosting company where we could get a little more server for our money.

As an aside, looking at the server hardware available in 2017 reinforced to us the importance of being prepared to move servers and not feeling like one is held hostage to the current hosting company. That’s because the way hosting works typically is that your monthly/yearly rate is locked in, so that after a decade you are paying the same amount for hardware whose value has dropped precipitously. Unless you are prepared to walk away from your current hosting contract you are doomed to overpay for your hosting costs over time, even if your initial contract was a bargain. I was determined that in the future we would not find ourselves back in our current situation where moving servers was so difficult that we had to continue to pay for 8 year old out-of-date hardware because moving was too difficult.

The LiquidWeb test server was actually the second test server I had set up. Before that I was using a temporary server generously made available to us by DC forum member PhilB. With that, I got some early practice setting up, organizing, configuring the CMS, forum, and the custom server software we need. I took notes as I did it and this practice in setting up a new server helped to ensure that it would be a much smoother process moving servers in the future (in fact our use of the commercial cPanel/WHM control panel software should make migration trivial if we simply want to create an exact duplicate of the server setup). I will return to this issue at the end of this essay.

To find an alternative hosting company I went back to a well regarded forum for such matters, webhostingtalk.com. I’ve always received good hosting advice there over the years, and after posting about an unpleasant bait-and-switch deal advertised by another company, I came into contact with the person running a hosting company named HiVelocity. After some research into the company and reading about other people’s experience with them, I decided that they would be a good match for us. Their prices were significantly lower than LiquidWeb, and their hardware options were a good match for us. I appreciated that there was a real human running the company that we could reach out to if we had to, and their focus on dedicated server hosting (rather than trying to cover the full spectrum of low-end shared hosting). While the initial setup process had a few hiccups, so far the performance and support has been great and I’m very happy with the decision to go with them. Of course the proof is in long term performance, so we will have to wait and see.

The other hard decision we had to make, besides which company to use, was deciding what hardware we needed for our server and and how much we could afford to pay. As for hardware, there were actually two related decisions that needed to be made. The first decision was what kind of hardware we needed to power the main DonationCoder.com website, and the second decision was how we wanted to set up the “member” server -- as a separate physical server, or running in a virtual machine on the same server as the main one.

These were hard decisions that would have serious ramifications going forward. I knew that the move to a CMS system was going to put an increased demand on the server resources compared to the old static pages, but hardware prices had dropped in the 8 years since our last hosting account, so I was confident that we could afford a server that could handle the increased demand.

More difficult was the decision about the member server. We still wanted to provide web space for programmers on our site to host their software -- in fact we wanted to do more in this regard going forward. In 2017, our existing setup had the member server and the main server running as virtual machines on one dedicated server. This had advantages and disadvantages. The major advantages were that we only needed one dedicated server and could still keep the member server and main server safely isolated from one another so that a security breach on one would not affect the other. The downsides were the increased resource use of a virtual machine, the additional maintenance headache and complication of the virtual machine layer, and the difficulty in balancing resource use across the virtual servers.

In the end I decided that the extra complexity of managing the virtual machines went against our mantra of simplifying management and had to go. This left two alternatives: Run everything on one server and risk a security breach bleeding over between accounts, or purchase multiple physical servers. While it would have been much simpler to run everything off the same server, I did not feel I could in good conscience do so. The point of the member server was to give accounts to friends of the site, and we simply could not guarantee the security of those accounts sufficiently to justify hosting them on the same server as the main DonationCoder content. For security purposes, I felt like we needed to keep the main site on its own server, isolated from any other user accounts.

So this meant purchasing two distinct hardware servers. In the end we were able to negotiate a deal with HiVelocity for two identical servers, for a total of $400/month. This was slightly more than we had been paying SoftLayer ($367/month) for our single server. The following is a summary of the costs from different hosting companies:

Our old (circa 2010) SoftLayer server was $367/month for:

• Dual Xeon 2.66GHz Intel Xeon-Harpertown (5430-Quadcore); 8gb ram; 500gb(?) sata hard drive; nominally gigabit uplink (but in actuality pretty slow upload speeds); unmanaged. On this server we ran VMWare ESX with virtual servers for the main DonationCoder.com server and for the secondary member server. Note that this price was set in 2010 and we would have gotten a better deal from SoftLayer had we wanted to stay with them (though they are one of the more expensive hosting companies and would not have been competitive with the other quotes we were getting; we originally decided to splurge on the SoftLayer server because we were desperate for a reliable hosting company).

Liquidweb’s best offer was $400/month for two (i.e. $200/month/each when paid a year in advance and committing to a 2-year plan):

• Intel Xeon E3-1230 v5 Quad-Core; 16gb ram; 2x500gb SSD (raid), 1tb sata; 5tb bandwidth; 100mb uplink.

Our new HiVelocity servers are $400/month for two (i.e $200/month/each when paid a year in advance):

• 3.8ghz quad-core E4-1270v6; 32gb ram; 480gb SSD drive, 1tb sata drive; 1000mb uplink (in practice much faster upload speeds than Softlayer);20tb bandwidth; 10gb cloud storage with daily remote backup plan; cPanel license; fully managed. One of these servers runs the main website, another runs the member server.

There can be little doubt that our new servers are overpowered for our current needs -- especially the member server which currently does relatively little traffic. And I can’t say that it’s not scary to have a $400/month server bill given our reliance on donations. But the new servers give us room to grow and I hope that we will be able to make more use of the member server in the near future for more spin-off projects and use the extra server power to be able to host some interesting new projects.

Changes and Improvements to the main DonationCoder Site

The main purpose of moving our web pages to a CMS was to make them easier to maintain and edit, and make it easier for us to add new pages and sections. But in the spirit of performing a major overhaul of the site I took the opportunity to make a few other changes in order to simplify and improve existing features. I will discuss two of the main ones below.

Simplifying the license key page and licensing process

One of the more confusing pages on our website has always been the page where people went to get a free license key for our larger software programs. The license key page tried (and often failed) to explain the process for getting a free permanent license key to our software, which was essentially:

• Users could get a permanent license key to all of our programs by making a donation of any amount.
• Users could sign up at our forum to get a 6 month license key, and then return in 6 months to get another 6 month license key and then return in 6 months to get a lifetime license key.
• Users who didn’t want to sign up at the forum could get a short-term (30 day) license key for a specific one of our programs by clicking on the name of the program they wanted the license key for.
• Users could email us if they fell into a special category (freeware/open source coder, non-profit company) for a lifetime license key.

While it may seem overly-complicated, the main idea of our licensing process was just to get people to consider the idea of donating, and not make it so easy to get a license key that people avoided donating just out of laziness in order to go with the fastest, simplest process. We hoped that by getting more people to register on our site, we would increase the quality of discussions on our forum. I wrote at length about our licensing process in my article on the early years of DonationCoder.com.

This licensing model has in many ways been a success. It has caused people to consider donating in a way that they wouldn’t if there was no license key system at all, while still letting people who can’t or don’t want to donate to use our software without limitation and for free.

But it has also resulted in confusion and frustration. The hardest part for me to continue to justify has been the idea of pushing people to sign up at the forum. I had hoped it would lead to an increase in community involvement and discussions, but this was probably mostly wishful thinking. The end result of the policy was lots of forum accounts of people who didn’t want anything to do with the forum, and some real frustration.

So with the server migration in 2018 we decided to make major changes to this process. In essence we changed the license key page to a simple multiple choice list, where people can choose the description that best matches their scenario. There is now an option to donate for a lifetime key, an option to instantly get a license key for all of our software that lasts for 60 days, and an option for anyone to submit a form to request a lifetime license key for all of our software without paying anything.

These changes in the license key procedure have resulted in a much simpler and clearer process, with much less confusion and frustration, and a much easier way for people to get a lifetime license key for free without donating. But at the same time it still requires someone to stop and think before they skip donating, and it gives us some quick feedback from those who request lifetime license keys. My fear was that it would drastically decrease the donations, but those fears appear to have been unfounded. If anything the donation rate seems to have gone up slightly, while at the same time, the new process seems to be much more pleasant and less confusing for people.

Back-end management tools

One of the major things that was done with the new site was to collect all of the disparate, custom back-end maintenance scripts that had been written over years and rewrite them to make them easier to use and maintain. Integrating them into the CMS had the added benefit of allowing us to enforce proper access control restrictions so that they could be used by anyone who has been delegated administration permissions for the CMS. The payment management and license key generation system was completely rewritten by me to operate in a much more streamlined way for donors and commercial license purchasers, with automatic forum account creation, etc., and a completely new back end management system that makes it easier for us to track donation statistics over time.

Creating a “Friends of DC” section

Over time we have struggled to find a good and consistent way to “host” software written by members of our community forum.

On the main site we have hosted the larger software programs written by me, as well as other software programs written by long-time trusted friends of the site. There have been a few problems with this approach. First, the old static web pages made it hard for anyone but me to edit these software pages, even just to announce a new program update. Second, it was hard to lay out clear guidelines for when we would add someone’s software to the main site; there was always a security risk in adding other software to the main site unless I was absolutely sure it was 100% safe. Third, it led to some confusion about favoritism and how we should share donations with these software programs.

Other software written by members of our forum (some long-term members, some not), was sometimes hosted on an author’s own web servers, sometimes on open source repositories, sometimes as forum attachments, and sometime on pages on our member server.

While we may revisit the issue, the solution we’ve elected to go with for now is to do a much better job of highlighting and “promoting” software pages of our members, and put them ALL on their own mini-sites on the member server. In this way, the main site now keeps a nice list of friends of the site who are coders and their associated web pages, and each coder can easily maintain their own software in their member server area. With this approach we hope to bring them some recognition and audience from the main site, while allowing them to maintain their own content and solicit donations directly.

The New Member Server Network

As discussed previously, soon after we moved to a dedicated server (2010) we began providing accounts to friends of the website -- long time forum members and coders, etc. By 2017 we had created accounts for over one hundred individuals.

These accounts were made on an ad hoc basis. Most were given FTP accounts and web space on our member server, some had their own domain names parked, and some were given shell access where they could compile software. Many installed their own blog systems or content management systems (WordPress was common), and were given mail accounts.

Over time, many of these web accounts were abandoned. Many of them had installed software that had not been updated with the latest security patches and thus presented a real security risk.

With our 2018 move I wanted to figure out a way we could continue to offer web space for our members -- most importantly for coders who hang out at our forum -- in a way that was easy to maintain and didn’t suffer from the security risks associated with having dozens of different software installations that had to be maintained by each user independently.

WordPress Multisite Network

In the end, the solution that best matched our needs was a multisite WordPress network.

While there are a large number of capable cms/blogging systems, WordPress includes some fairly unique functionality that allows a single install to host a large “network” of individual user sites. This is the so-called “multisite network” setup. Each user site is in effect a separate blog/website, which shares the resources (themes, plugins) of the main site, and can be managed by one or more administrators. There is only one single central WordPress framework installation which needs to be maintained and configured and kept up to date, making maintenance trivial.

A third-party duplicator plugin lets us create new member sites with preconfigured options easily. In this way we can quickly create a new member site on the network that is setup and ready to go with a software download page, contact form, etc. Each user is the administrator of their own site and can customize it thoroughly choosing their own theme (from a list of network administrator installed themes) and options, and they can maintain their own content. While the look of each site can be customized, the sites tend to share some common appearance elements which helps tie them together into a unified collection.

Using the WordPress multisite network we can easily provide any friend of DC with their own independent site where they can upload their software, make a website, and blog. Furthermore, we have a central page where we have links to all of our member sites and an RSS feed that displays new content from any member page. This solution allows us to present the member sites as part of a collective.

By limiting member accounts to this multisite WordPress network, we are imposing constraints on them. They cannot install arbitrary scripts and cannot install their own content management software. The loss of some freedom was a necessary concession to allow us to safely and easily maintain the new server going forward. Moving all of the old content was labor intensive and was done by the members themselves with some help from me and others (DonationCoder member Skwire in particular).

Easing DNS Changes with Cloudflare

When a domain name is registered, one specifies DNS nameservers which tell people what physical server to use to resolve requests for the website content. When we had previously moved servers in 2010, the general process for moving servers was to set up the new server, then change the DNS servers at the registrar (where the domain name is registered) to point to the new server.

A major problem with this approach is that such changes can take up to 48 hours to propagate to all internet users and third-party websites. This is a problem because it means there can be a 48 hour window where some people are going to the old site and some to the new site. If the old site is down these people get broken pages. If it’s up, they get out of date older pages.

When we moved servers in January 2018, we used a different approach which improved the process dramatically. Several weeks before the server move, we created an account at Cloudflare.com, which is a DNS proxy service. While it has some advanced features, including some paid features, we used only the most basic features that allow Cloudflare to act as a DNS name server middle-man and allow you to quickly change the server ip for a domain name. We set up our Cloudflare account for both our main and member server to point to our old servers, and changed the nameservers at our registrar. After 24-48 hours this change had propagated to the internet, though no outward change was visible to users -- the domain names were still pointing to our old server.

With that done, when we were finally ready to change over to the new server, it was a simple matter of changing the Cloudflare configuration. For all intents and purposes all internet users were now immediately switched over to the new server without significant delay. An additional useful feature of Cloudflare was that it let us redirect certain subdomains of the member server to the old server and certain ones to the new member server. That allowed us to migrate subdomains to the new server as we finished them, while keeping unfinished ones live on the older server. It also would have let us easily abort the move and switch back to the old server had something gone unexpectedly wrong.

Conclusion

The key lesson from the story of the DonationCoder.com infrastructure remodel (moving servers, moving content to a CMS, reorganizing the member server), is how important it is to design things for long term maintenance.

As a programmer this is one of the most important concepts to learn, and it requires constant vigilance and discipline. Part of the problem is that with many projects it’s not at all clear that you’re going to be creating something that has to last 10, 20, 30 years on, and the demands to get things working quickly are usually foremost in one’s mind. Sometimes the substantial extra time required to design something with maintainability in mind can be hard to justify. However, there are principles that can help keep things maintainable. One of those principles is to always force yourself to explicitly consider both the immediate costs of a design decision, and then the future costs to maintaining that decision. For example, there are many occasions where one is tempted to customize some piece of third-party code in an existing framework, or customize some non-standard option. The immediate costs are negligible and the result is a more customized solution that your users want. However, one must consider the long-term maintenance costs -- even if it’s just the cost in terms of added complexity of keeping track of all the customizations that will have to be migrated to a new version.

Another technique for keeping your focus on maintenance costs, and reducing long-term eventual maintenance costs is to practice migration. This is the same principle that people practice when it comes to system backups. When you put in place a new backup system, you don’t simply run the backup and assume that the restoration procedure will work, you perform a dry run and make sure that the restoration works and can be used to recover the system. In the same way, periodically practicing migrating a server can be extremely valuable in terms of forcing you to keep good notes about the configurations used, the gaps in your records, etc. In fact during the course of migrating the DonationCoder servers I performed the basic procedure twice before our actual move, each time finding new small items that I had failed to document in the previous attempt. In the same way, periodically reinstalling your home operating system has similar benefits -- it helps you establish and test a procedure by which you can recreate your working environment cleanly and reliably, so that if and when you find you need to do it for real it is a process you are prepared for.

Finally, a word about the hunt for the perfect content management system. It doesn’t exist. At a certain point you must simply choose, among those with a healthy developer community, the tool that best matches your anticipated needs, and which can be cleanly extended by you to provide any missing features that you need. And remember that If your project survives long enough, you may well eventually have to migrate to a different system, so keep notes and plan accordingly.