Technology: Backing Up to Move Forward

Shortly after 3 a.m. on Sept. 24, 2005, Hurricane Rita’s 20-mile-wide evil eye hit the coastline at the Texas and Louisiana border, killing 120 people, wiping 100,000 homes off the map, and uprooting 1.5 million trees.

“For the first time in my adult life, I experienced fear,” admits James Holly, a family physician and chief executive of Southeast Texas Medical Associates, or SETMA. “I’m accustomed to changing things, fixing things. But with this, there was nothing I could do.”

True, he could do nothing about the storm. Nevertheless, he and his 23-physician practice were prepared to deal with just such a catastrophic event. Only four days after the storm, SETMA reopened its Beaumont, Texas, clinic, and it had the EMR operating later the same week.

Not that there were many takers. Ninety-four percent of the greater Beaumont residents had evacuated. “We were so proud of ourselves because we were open, and I think we saw eight people,” he says, chuckling.

Although the survival of patient records may seem relatively inconsequential to the general public in the aftermath of a fatal storm, doctors know better. Accurate and accessible records are critical to the health of your patients and your practice. It doesn’t take an historic hurricane to wipe out your EMR, either. Virus attacks, power outages, hardware or software failures, and human errors are all powerful enough to bring your practice to its knees.

If disaster strikes, a clear recovery plan will make the critical difference between bouncing back and jeopardizing everything you’ve worked for. Are you prepared?

Failure to plan

At an absolute minimum, doctors should be backing up data on tape, which is the most common method. Back-up tapes should be taken offsite for storage - to a bank vault, for example. Storing tapes at home can save you time, but a storm could claim your office and home in one fell swoop.

Not surprisingly, taping isn’t foolproof. The tape itself could fail when you need it, says James Morrow, a family physician with North Fulton Family Medicine, a four-clinic practice in the Atlanta area. Tapes are at risk for accidental deactivation. They can be lost. Or there can be a recording failure. “[Practices] often use VHS. But then when they put in the tape, they’ve got ‘The Young and the Restless,’” Morrow says. “There should be a fire siren going off in the server room if something goes wrong, like a backup failed.”

David Ott, vice president of technical services for EMR vendor NextGen, concurs: “We had one customer who was in the process of upgrading. It turned out they had not had a successful [backup] for a year. They thought they were backing up every day, but were actually backing up the same file every day.”

Other lapses can result in disaster, too. North Fulton’s practice leaders learned this the hard way. “When we got infected with the Klez virus in 2002,” Morrow says, “we were under the impression that [our vendor was] maintaining virus protection, and they were under the impression that they were not.”

The group couldn’t prove anything. “We didn’t have it in writing, so we just had to suffer through the old ‘he said/she said.’”

North Fulton had to manually enter about 500 patient visits, replace the server, pay staff overtime, and endure nearly three full days of downtime. Total cost? About $25,000. “That was the impetus to change,” he says.

Who should do the backing up?

Your EMR vendor probably offers disaster recovery, perhaps through a third party. For example, NextGen offers its WellCare program through Silverback Technology. WellCare responds to disk crashes, ensures successful data backup, scrutinizes computer use to guard against virus attacks, and constantly scans and reports on the server’s status.

If trouble of any kind is detected, the system immediately sounds the alarm. “We’ll get a message - ‘The backup has failed’ or ‘Low on space’ - so we’ll call the client and find out what’s going on. It raises the level of awareness,” says Ott.

The cost for using WellCare? Zip. “We put it in place to prevent a forest fire,” Ott explains. “It’s free, but people are always skeptical of free, to a point where we even considered charging a little fee for it.”

Some of you may feel uncomfortable with allowing an outside party to handle your backup needs. After all, we’re talking about private patient data. “They’re suspicious that we’re monitoring their activities, bleeding data,” says Ott. But there’s no need to worry, he says. “We don’t actually have access to your system; it’s all event-driven. We react to messages your system sends.”

Ott says NextGen makes a tremendous effort to apprise its clients of WellCare’s capabilities. “We can’t make you do it, but we can make you aware and encourage you to do it.”

North Fulton Family Medicine takes advantage of its EMR vendor’s backup service, Healthmatics Secure, which is supported by Allscripts. “It’s the reason I can sleep well at night,” Morrow says. “Every hour, our data is backed up to a server in Austin, Texas. If we have a downtime where we can’t access the server, we can tap into the second server.”

The practice’s Alpharetta, Ga., office tends to lose power regularly owing to area construction. Morrow called the EMR vendor, Allscripts. “I said I didn’t want to alter data on the [secondary] server, but to be able to get information - sort of a read-only.”

Fifteen minutes later, the practice was up and running.

Still, you may choose to set up backup/recovery procedures yourself, particularly if you’re tech savvy.

SETMA established its backup plan a decade ago, before its EMR vendor, NextGen, offered WellCare. The practice is loath to retrofit its protocols, but not because NextGen’s solution is lacking. “We’ve done a lot of customization,” says Holly. “The Internet was not what it is today. Once you start on a course, it’s hard to change it.”

North Fulton Family Medicine outsourced its backup needs to an IT company until about three years ago. “When we finally got to the point where we were writing a check every month equal to what we would pay someone in-house, that’s when we went in-house,” says Morrow.

So what’s the plan?

Redundancy is key, says Ott. “When we do an implementation, we require - or at least strongly recommend - that you build redundancy with your implementation,” he says. “Computers are what they are, and you should build in redundancy to guard against failure.”

This means having a backup plan for your backup plan, such as a secondary server - preferably offsite - that holds what is essentially a mirror image of your live data. The routine backup procedures you set up in your office should write to this server on a regular basis.

And what’s “regular”? Something different for every practice. For SETMA, it means every 15 minutes. “The maximum [amount of data] we can lose is 14.5 minutes - fairly inconsequential,” says Holly. His practice also backs up the entire system twice a day to tapes, stashing them in a bank vault.

SETMA’s backup is asynchronous, meaning the data on the two servers are never exactly alike. Synchronous backup occurs when the system backs up every time a transaction is entered. Obviously, the more often you back up, the less data you stand to lose. However, synchronous backup slows down system performance, which is annoying to EMR users, and it’s quite expensive, which is annoying to whoever has to pay for it - probably you.

A written plan is also crucial. When disaster struck, Holly says SETMA invoked an automatic order that called for Dell to ship a new server to NextGen to reinstall the EMR software, and for a bank to run a line of credit “before any insurance involvement so we [could] get moving again.”

Here Holly means “practice interruption insurance.” He says, “I would encourage any practice to have that. Our practice lost about $1 million in revenue, as ongoing expenses were not arrested. The bank said we didn’t have to pay for 90 days, but eventually you have to. Practice insurance kept us going. There was a time I was worried, but it worked out.”

Finally, make sure that your backed-up data are encrypted before transit. Encrypted data are “scrambled” in a way that prevents outsiders from being able to read it. This is critical to protect the privacy of your patients’ information.

Shoulda-coulda-woulda

Unfortunately, no matter how thoughtfully and carefully you devise your backup and recovery plan, you’ll always discover something you wish you had done before disaster hit. Even Holly, whose practice has a comprehensive strategy, wishes he’d had a generator to run the network locally. “I’d like to have that so we could be up and live during the whole thing,” he says.

Holly feels good about his disaster recovery plan, but he’s definitely not bragging. “I don’t want to sound arrogant,” he says. “We’re certainly not immune. But we’re very careful. We’re dealing with life-and-death issues.”

And anything can happen. An air conditioning unit once failed in SETMA’s server room, catching everybody by surprise. “When the temperature got to a certain level, the system shut down. We realized then that the redundancy we were so keen on should include the A/C. So we’re putting one in, plus an alarm,” Holly says. “A system must be analyzed at every level for those single points of failure.”

Morrow says that his practice faced a reality check, too, after suffering through the Klez virus attack. “I thought we were good then. But you can’t know what you don’t know,” he says. “If there is a potential fault in our system, it would be in being sure that the data is being sent regularly. There are people being paid to make sure it is. But just because we’re signed up with Healthmatics Secure, it is not enough. You have to be sure. You have to check it, and we do.”

In short, don’t get complacent about your backup system, even after it’s in place and running well. Check it regularly, just as you would the oil in your car. If not, Morrow recommends “bigger bottles of Valium, I guess.”

Shirley Grace, senior writer for Physicians Practice, holds an MA in nonfiction writing from The Johns Hopkins University. Her articles have appeared in numerous publications, including The Washington Post and Notre Dame Business magazine. She can be reached at sgrace@physicianspractice.com.

This article originally appeared in the June 2007 issue of Physicians Practice.