Backup and Disaster Recovery

The Scene

Monday, 7:03 a.m. You arrive at the practice to find your staff a little more animated than usual. One of the building's restrooms above has flooded overnight, and water has been cascading down into your server room for the last several hours. Total outage: Your network is down, computers cannot log in, no email, no EMR, no billing. Frantically, you call your IT vendor, who arrives onsite to proclaim that most of the computer equipment in the server room is ruined, including the tape drive that backs up each night and the stack of backup tapes kept alongside it. A bad scene that could be repeated for any and all business-altering situations: fire, burglary, hardware failure, human error. You wonder out loud about your insurance coverage for replacement hardware, about your backups. Who is in charge of the disaster recovery? How many hours or days will your practice be down? What about HIPAA Title 2?

Business Continuity Impact

Like any business, your practice has financial responsibilities contingent on keeping your doors open, patients flowing, and turning a profit. Any adverse impact on the ability to make a profit is noteworthy. A data outage, large or small, can be one of those impactful instances. Consider the yearly gross revenue for your practice and divide that number by 223 days. Let's assume $1 million gross divided by said amount of working days: about $4,500 of potential gross revenue lost per day during an outage if ALL staff are unable to be productive and are sent home - a disaster to be sure. To get a more refined figure per person/hour, divide the daily gross of $4,500 by your number of staff. Very soon you get to see the true, direct financial impact of an outage. We say "direct" because there are indirect costs, such as the need for additional staff to "catch up" when the system is available again, patient dissatisfaction, and the ripple effect of things missed, appointments dropped, files gone, and schedules delayed. Time to ask questions.

Risk Assessment

Backup and disaster recovery planning are strategies to mitigate risks. Correctly identifying and assessing those risks and their impact is critical to the survival of your practice, your business. Backup and disaster recovery planning are business issues, not technical issues. The word "backup" means many things to many people. For some it is simply the ability to recover that missing or overwritten Excel spreadsheet you spent weeks developing. To others it is something nebulous that is assumed to be taken care of by their IT provider.

Mitigating Risks

What format is your mission-critical data in? Paper charts? Scanned documents? Transcription files? Imaging files? Full-blown EMR system? Where is your mission-critical data? Onsite client/server? In the cloud on the Internet? A hybrid of both? What is the hierarchy and communications structure in your practice in the event of an outage or data loss? What policies and procedures exist, and who is to implement them? How long can the practice survive before the finances are significantly depleted, thereby threatening the very existence of the business?

In HIPAA jargon your practice is a "covered entity." HIPAA Title 2, referred to as "administrative simplification," includes provisions for "the implementation of controls to protect an individual's health information." What about destruction and disposal of damaged hardware that may contain patient information? In a recent study by Pepperdine University, the risks associated with data loss include:

• hardware failure (40 percent)
• human error (29 percent)
• software corruption (13 percent)
• theft (9 percent)
• computer viruses or malware (6 percent)
• hardware destruction (3 percent)

According to the U.S. Bureau of Labor, 43 percent of companies that suffer large-scale data loss due to disasters never reopen, and 29 percent close within two years. Additionally, 93 percent of all companies that experience "significant data loss" are out of business within five years. Risks will exist regardless. Now the strategy of what risks to mitigate and what recovery procedures to have in place.

What to Back Up

You cannot electronically back up what you do not have stored electronically. Conversely, you cannot electronically recover lost data that you have not backed up. Carefully consider what to backup. The space taken up by the 200 pictures from last year's holiday party or an employee's 4GB iTunes collection all of a sudden pale in comparison to the need for backing up X-ray imaging, scheduling calendars, billing data, payroll data, EMR databases, and email mailboxes.

Work with your staff to uncover the most important aspects of your data. Work with your technology vendor to find out what gets stored, where it gets stored, and what you will need recovered in a true disaster situation. Create an archive process for information that has been deemed unneeded for regular or frequent business use. Archiving will effectively and in a controlled manner remove data from the day-to-day storage and render into long-term storage. This will save both time and money when coming up with infrastructure solutions to meet your regular backup needs and your disaster recovery planning.

Business Recovery Considerations

Planning for disaster recovery depends in what business thresholds you set your practice to sustain. In other words, how many hours or days can your practice afford to be down? In IT circles there are two concepts to consider: recovery time objective (RTO) and recovery point objective (RPO). RTO is how long your practice can go without a specific application, e.g., email or EMR. This is often associated with your maximum tolerable outage. If your RTO is zero (cannot go down), then you may opt to have a completely redundant infrastructure with replicated data offsite, online, and onsite. If your RTO is 48 or 72 hours, then perhaps a less expensive, less redundant infrastructure will suffice.

The RPO dictates the allowable data loss - how much data can your practice afford to lose? In other words, if your practice performs a nightly backup at 7 p.m., and the system crashes at 4 p.m. the following day, everything that was changed since the last backup is lost. The RPO in this particular context is the previous day's backup. In larger practices that may perform many transactions per hour, the RPO should be down to the last, latest transaction that came in lest an entire day's transactions have to be reentered. Determining your tolerable business thresholds will directly affect RTO and RPO levels and associated infrastructure costs. You can safely surmise that lower values for RTO and RPO are directly proportional to higher implementation costs, so plan accordingly.

How to Back It Up

Historically, tape has been the default means of taking a system backup, especially where large amounts of data, possibly housed on a server, were concerned. Although tape arguably still maintains the best cost-per-gigabyte up-front cost, there are inherent risks to tape media handling, such as damage or, even worse, media loss. The latter has been plaguing many companies in the financial sector, giving newer technologies a clear advantage. As a matter of fact, we no longer recommend tape backup in new or updated installations. There are alternative technologies that offer faster backups and more reliable restores. Among them:

• Disk or Disk Array: The price of disk drive storage has come way down. Hard drive technology tends to be much faster, more reliable, and less susceptible to environmental effects. Large hard drive storage negates the need to use multiple tape media to store a single backup.
• Image-based: This newer backup method is quicker, compresses much better, and, when combined with disk media or online storage, allows for a more flexible backup scheme.
• Online Storage: Businesses are embracing the Internet as yet another tool for backup, allowing for a method to automatically push offsite backup images out and store them in geographically disparate places without the need for human interaction and very few dependencies.
• Hybrid: A combination of the above three technologies. A backup solution can be designed that is able to take data snapshots at very frequent intervals — 15 minutes — while your practice continues to operate normally.

Consult with your technology vendor on what you do (or don't) have in place now and how you can leverage the above technologies.

What About Insurance

Insurance considerations are topics onto themselves, but, for purposes of disaster recovery planning, make sure your insurance agent reviews the following coverage types and inclusions:

• inventory and catalog all IT components (HIPAA)
• copy to insurance agent/head office
• inland marine coverage
• hardware replacement costs plus reinstallation labor
• flood insurance
• filed class and non-filed class coverage
• business interruption coverage
• sprinkler systems and smoke alarms
• theft vs. burglary
• user alarm codes and monitoring
• additional umbrella policies as needed

A good insurance policy that includes the above provisions is part and parcel of an effective overall disaster recovery plan. It is essential when bringing the IT infrastructure back to normal as quickly as possible.

The Bottom Line

Simply stated, statistically, your business will not make it in the event of an outage or significant data loss if you do not have a backup and disaster recovery plan in place. Protect your practice from becoming a statistic by designing and implementing a disaster recovery plan that includes risk identification, comprehensive backup, hierarchical communications, insurance coverage, and overall common sense. Be sure to check with your IT provider regarding safe backups, practice runs, test backups, and restores. Once your plan is complete, then have staff practice it, maintain it, and perfect it. A good plan clearly states what needs to be done, who needs to do it, where, and when.