IPMA Home | IPMA News IPMA News : February 2004 Edited By Mary Ellen Bradley and Dennis Jones

SAO Results of IT Compliance Audits

DIS Facilitates Disaster Recovery Review

DFI - Disaster Recovery Planning - Lessons Learned

Yakima County Secure Data Center

News from the Past

Summary of January 29-30, 2004 IPMA Board Annual Planning Meeting

SAO Results of IT Compliance Audits

-- by Pete Donnell, CPA, Audit Manager, Statewide Technology Audit Team, Washington State Auditor's Office

The State Auditor’s Office has performed about 40 audits of agency Information Technology Security Programs for compliance with the Information Services Board’s (ISB) Information Technology Security policies and standards. The majority of these were performed in September of 2003 We are still performing a few of these audits for agencies that needed a little more time to complete their programs.

We believe most agencies made a strong effort to develop a program that met the requirements outlined in the ISB policies and standards. During the course of our audits, we determined most agencies found some value in updating security programs. We also noted many agencies had security practices in place that were not documented by policy or other written form.

The following table presents a summary of recommendations we made as a result of our audits. The numbers relate to recommendations made and not to the number of agencies with recommendations. Recommendations classified as exceptions relate to conditions of non-compliance. We also made suggestions that, in our opinion, could improve security programs in a few areas, even though overall, the area was considered to be in compliance. It should be noted we were auditing for over 250 items in the following areas.

In addition to presenting the table above, it is worth mentioning a few of the common deficiencies we found. Agencies did not always have an IT security strategy overview section. Some did not have documented risk assessments for business applications and Internet facing applications. A number of agencies had performed risk analyses related to locations and disaster recovery. However, the risk analyses did not distinguish different security aspects that might be needed because one application has more critical information than another. A lot of agencies overlooked documenting risk analyses for their web pages. Agencies seemed to document only those web pages related to e-commerce.

In the data security area, many agencies did not address the typical accounting controls related to ensuring the accuracy and completeness of data entry. In the access security area, common deficiencies were a lack of hardened passwords and/or not having lockouts occur after five bad log-on attempts. Agencies also struggled with Section I.E.4, Network and Telecommunications Security. This section contained requirements that were addressed in other sections of the standards. A common oversight was in documenting how access to the network was controlled. For example, many agencies did not address who would approve access and how access would be limited to only the level needed to perform job duties.

We also noted that many programs would tend to present how the agency was doing things rather than present a policy or standard for how things were to be done. We made a number of suggestions for agencies regarding the tone of their policies. Security programs should set the standard of what is required and allow for exceptions or exemptions. The exception or exemption process should require documentation that presents compensating controls or document other factors that will not put the agency at risk due to the exception.

Lastly, we also noted most agencies had not updated their program to reflect the requirements of the new standards issued in March 2003.

While we made a number of recommendations, we do think agencies overall did a good job at developing a written security program. Agencies generally received 10 to 20 recommendations. Given that we were looking for over 250 items, this is a good showing for this first round of security audits.

Return to Top

DIS Facilitates Disaster Recovery Review

-- by Lance M. Calisch, Department of Information Serivces

DIS has a current disaster recovery (D/R) program in place that was originally established to meet the needs of the mainframe applications. Because the needs of the agency mission critical systems have grown to include many server-based implementations DIS needed to review the current Business Continuity and D/R requirements in the community.

In December 2002, DIS commissioned the services of an independent vendor to review its disaster recovery (D/R) program. It had been approximately eleven years since the inception of DIS’ D/R program and DIS wanted to see how the program was meeting the needs of its customers. In addition to an internal review of its program, DIS asked the vendor to interview twenty customer agencies. The conclusions from the internal review and the customer interviews are summarized below.

First, the DIS mainframe D/R strategy is solid. This program uses an external hot site provider and is tested twice per year. Participating customers exercise their ability to access DIS mainframe systems recovered at the hot site. The Recovery Time Objective (RTO) for DIS mainframe services is 72 hours and the Recovery Point Objective (RPO) is 24 hours. RTO is the maximum time it takes to recover the mainframe environment from the time of disaster. In the event of a declared disaster, DIS would recover the mainframe environment ready for customer use within 72 hours. At the time of recovery the maximum data loss objective (RPO) would be 24 hours prior to the disaster.

Second, the existing statewide network infrastructure could not survive a significant catastrophic event. The goal is to architect a network that can withstand the complete elimination of any single node site. DIS is in the process or has completed efforts to make critical network components redundant. A second Internet service provider (ISP) connection was completed in May 2003, redundant core network switching was completed in August 2003, and redundant domain name service (DNS) is underway. Redundant components for Active Directory, Access Washington, Transact Washington, and Fortress are targeted for completion by June 2005.

Third, D/R services are not currently available for DIS hosted server technology. DIS is reviewing options for recovery of its hosted distributed server technology environment.

Fourth, no meaningful intrastate recovery strategy existed. DIS is investigating suitable recovery strategies for locations in eastern Washington. This includes data center alternatives, the potential use of existing public sector facilities (e.g. Yakima County’s Secure Data Center), and establishing links with local government entities. DIS is working with Yakima County to position the DIS network to support SGN customers at the Yakima County Secure Data Center.

Lastly, customers asked for a facilitated network opportunity with others. The objective of these facilitated focus groups is to exchange shared knowledge and resources. For instance, if agency A has excess floor space and agency B is limited on floor space then they may be able to help one another. Another opportunity might be agency A, B, and C coordinate resources to share a D/R facility. There are two basic objectives to these meetings; one, share information and two, share resources.

On November 7, 2003, DIS facilitated the first session designed to engage agencies in a collaborative dialogue concerning their disaster recovery needs. The event was well attended and well received. Twenty-one representatives from thirteen state agencies and one county were in attendance. Lance Calisch from DIS facilitated this first session and a second is planned for Friday, March 5 from 9 AM to 11 AM in the DIS Academy Classroom. If you wish to attend please contact Lance at LanceC@dis.wa.gov or call 360-902-3440.

Return to Top

DFI - Disaster Recovery Planning - Lessons Learned

-- by Ron Seymour, Department of Financial Institutions

The Department of Financial Institutions (DFI) has 148 FTEs that regulate a variety of financial service providers, such as banks, credit unions, mortgage brokers, consumer loan companies, and securities issuers and salespersons.

In December 2002, using Department of General Administration contracts, DFI hired Sierra Systems to develop a Disaster Recovery/Business Continuity Plan. The project followed a standard methodology for its development. The steps involved in the development processes were: project initiation, business impact analysis, business continuity plan development, testing/training, and plan maintenance. Although the project was formally completed in April 2003, maintenance and updates are in an evolutionary process. The development of the plan was a major agency wide effort involving everyone from the Agency Director to many of the line staff. All the Division Directors were involved at one point or another and the program managers had to make a significant time commitments to this project. It was also important to have a person whose full time job was to lead this effort to completion. In DFI’s case, this meant hiring a contractor.  The planning process, at times, seemed grueling but in hindsight it was also rewarding and an eye opening experience. The process brought visibility to many issues such as: the way we perform our systems backups may not be appropriate for rapid recovery, and simple things like missing phone numbers. There were many lessons learned during and after the process completion that will be discussed later in this article.  

The following is a brief description of the project phases.

Project Initiation: Laid the foundation for the Disaster Recovery /Business Continuity Plan and created a project charter that included: project goals and objectives, scope, project tasks, project organization, roles and responsibilities, milestones, assumptions, and a project schedule.   

Business Impact Analysis: We identified the business functions, ranked them in order of importance, established recovery time objectives, identified dependencies, and performed a risk analysis.

Business Continuity Plan Development: Describes what the agency is going to do in the event of a disaster.  It describes roles and responsibilities, emergency preparedness, incident response, facility recovery, Information Services recovery, and business contingency plans. For each business function an interim and full recovery plan was developed for getting each function back up and running following a disaster.

Testing/Training: Originally this was specified as performing a desktop recovery drill. What it proved out to be was a very good training exercise.  As we began walking through the scenario of a building fire it was apparent that even though people had been involved with developing the plan, it had not hit home about what the plan would do for us until we started acting out the roles. This was an educational experience for all involved. So we changed this phase from testing to testing/training.

Plan Maintenance: Included maintenance drivers, maintenance events, roles and responsibilities, document distribution, change management, and the maintenance schedule.

This illustration depicts the scenario that the DFI disaster recovery plan was based upon.

Lessons Learned:

Scope: We found out very quickly that if we did not define the scope of the disaster we were preparing for, all conversation became "What if?" scenarios and it was very difficult to accomplish the tasks at hand. So we decided that we would focus on the loss of our primary work location due to a fire. One of the assumptions that we made was that if there was a catastrophic event that disabled most state agencies in Thurston County, that because of the nature of our work we would not be a priority for recovering our services right away. This helped during the planning process, though it also presented some issues when we had to deal with an event that will be described later.

Information Technology: As we were developing the IT recovery plan it became apparent that if we were to lose our computer room and all its equipment, our recovery would have to be in a sequential order because we only use one backup system and set of tapes.  So we would only be able to restore one service at a time.  Estimating the time to recovery from backups we would not be able to meet the recovery time objectives of our business units.  The IS staff decided that it would be more practical to focus on continuity (keeping things running) and using recovery as a plan B.  Using the "A La Carte" services provided by DIS, we placed backup servers for our production environment at DIS.  These servers perform real time replication of our data.  In the event of loss of our production servers we can quickly switch to the backup servers with minimal if any loss of data.  We also established a high speed connection between our Seattle office and both DIS and our Headquarters office in Tumwater. With this configuration the Seattle office can be used as a hot site if we were to lose the Tumwater facility. We are also migrating our backups to DIS which has a more versatile system than DFI has. 

When the Blaster Virus Hit: As stated above, we used a building fire as the basis for our disaster recovery planning. However, in actuality, computer viruses and network component failures have had the largest impact on our business this past year. Loss of revenue, though important, was not a key issue in determining priorities of business functions; the rationale being, that we would make it up later. The nature of our business is fee driven and we can collect the fees at a later time.

Approximately, one third of our agency staff works in the field examining financial institutions.  The assumption was that if we had a building fire, they could keep working or if it was a catastrophic emergency there would not be a significant impact from failure to perform examinations in the short term.

In July 2003, the Blaster computer virus hit.  Staff who work on billable schedules were shut down because they didn’t have the latest patches on their laptop computers.  The patches provided by Microsoft were too large to download over slow dial-in connections and work stopped.  Now a business function (examinations) that had a 10-day recovery time objective was stopped and money was being lost. The IT staff that were scrambling to patch all the systems used the recovery time objectives to prioritize their work. For the program that had their people out of work, this was not acceptable. They were losing money and incurring travel costs to have their staff bring their laptops into a DFI facility to get their patches updated.

In the spring of 2004, when we update and retest our Disaster Recovery plan we will rethink the recovery time objectives and the disaster scenarios. 

What We Are Doing Differently?

• We have created new emergency handbooks for all DFI staff.
  
  
• We have an extensive list of phone numbers that may be need in case of emergency.
  
  
• As mentioned above we have reconfigured our network and production server environment to achieve better performance in a disaster.
  
  
• We have developed a process for patch management of all our servers and PCs.
  
  
• We have begun imaging many of our paper documents to mitigate the impacts of a disaster. 
  
  
• We will rethink our disaster scenarios before our next testing period in the Spring of 2004.

For more information you can contact Ron Seymour at (360) 902-8788 or email Rseymour@dfi.wa.gov.

Return to Top

Yakima County Secure Data Center

-- by Michael A. Sloon, Secure Data Center Manager, Yakima County Technology Services

Disaster Recovery/Business Continuity

Yakima County is now able to provide Local and State Governmental agencies with Disaster Recovery and Business Continuity Services through the new Yakima County Secure Data Center (SDC). The journey we took to achieve this goal began 4 years ago with the belief that the increasing Disaster Recovery and Business Continuity needs of local, regional, and state agencies could be resolved with a "collaborative government" solution.

Hearing that many governmental IT professionals through-out the state were searching for a cost effective solution to their Disaster Recovery and Business Continuity issues, we realized the location of our new data center (incorporated in a remodel of an existing county building) would accommodate a facility in which Disaster Recovery and Business Continuity Services could be built. In 2000 we began asking other governmental agencies the question: "If we built a facility in Yakima for Disaster Recovery and Business Continuity that was affordable, well designed, and secure would you use it?" The number of agencies who said yes, or expressed a "positive interest" was astonishing.

With assistance from The Gartner Group, State and Local IT professionals we began the process to build the new facility. In December of 2003, Yakima County moved into the new data center and "made-ready" the Disaster Recovery and Business Continuity services. Through the combined efforts of many individuals, agencies, and vendors this "collaborative government" solution is now a reality.

Background/Important Features

• Location. Yakima County sits upon the North American continental plate. This plate is solid basalt, averages 16,000’ in depth and is extremely stable geologically. It shares the same seismic characteristics found throughout Eastern Washington and the Northern Idaho Panhandle.
  
  
• Accessible. Yakima is less than four hours by car from most locations in Washington, and has an airport that can handle the largest military aircraft. Interstate 82 bisects the county from Ellensburg to Prosser. The SDC is located within a quarter mile of a main Interstate 82 off-ramp.
  
  
• Secure. The SDC is located within an existing fenced County facility that is both secure and seismically stable. The facility was highly rated in an evaluation by structural engineers; in the unlikely event that Yakima would experience a quake, the engineers stated, "…this would be the last building standing in Yakima."
  
  
• Uninterrupted Power. The SDC is located between two separate power grids that both feed the facility. A generator of sufficient capacity to power the entire county complex is installed and supplies conditioned power if utility power is interrupted. In addition, a 130 KVA Liebert UPS is installed and provides uninterruptible power for the first phase of the SDC.
  
  
• Environmental Control. The requirements of the SDC (temperature, air filtering, and humidity) are monitored and maintained with redundant HVAC units (with auto fail-over) engineered to meet the demands of the entire facility.
  
  
• Other Facilities. To support Disaster Recovery/Business Continuity services, the SDC provides lavatory facilities, showers, and a "war room" office that is wired for network connections and phones.
  
  
• Communication Hub. Yakima County is located in the hub of several communication carriers. Maximum redundancy has been created by optioning space on some of these data paths.

Service Options

• On-line Production / Remote Data Center:
  Server/s on-line and supporting a production environment. Facility would function as a primary or secondary site. Level of SDC service/s to be negotiated.
  
  
• On-line Idle:
  Server/s operational but in stand-by mode. Failure at the primary end would cause redundant system to come on-line to meet service demand.
  
  
• Near Line Storage:
  Customer data will be sent to a storage device/system for archival/backup needs and is accessible on-line.
  
  
• Offline Storage:
  Customer data will be sent to a storage device/system for backup needs.
  
  
• Hot-site Facility:
  In the event of a facility loss, customer would contract for a wide array of services. These services could include:
  • Office space with connectivity to business systems
  • Telephony support
  • Meals, shower facilities, and other accommodations on site
  • Staged setup of equipment shipped by vendor when a disaster is declared
  • Local technical support
  
  
• Other services being considered:
  • Web Hosting
  • Application Service Provision
  • Primary Data Center

Return to Top

News from the Past

5 Years Ago -- February 1999 IPMA Newsletter

10 Years Ago -- February 1994 IPMA Newsletter

15 Years Ago -- February 1989 IPMA Newsletter

20 Years Ago -- February 1984 Association of Data Processing Managers Newsletter

25 Years Ago -- February 1979 Association of Data Processing Managers Newsletter

Return to Top

Summary of January 29-30, 2004 IPMA Board Annual Planning Meeting

Members Present: Jim Albert, Mary Ellen Bradley, Thomas Bynum, Phil Grigg, Sheryl Hall, Dennis Jones, Dennis Laine, Andy Marcelia, Mike McVicker, Christy Ridout, Darrel Riffe, and Shelagh Taylor. Phil Coates, CFO was also present.

Mike McVicker, IPMA Chair, opened the 2004 Board planning meeting January 29, 2004, at 1:10 p.m.

By-Laws Review:

• Mary Ellen Bradley, chair of the Communications Committee, reported that several members of county/city governments have signed up for IPMA membership. The current By-Laws in Article V, section 1 states "Current or retired Washington State government employees are eligible for associate membership." The question before the board was should that section of the by-laws be changed to include county/city employees? The board determined that the by-law should not be changed and that the membership page on the IPMA web site be changed so requirement for state government employment is clearer. The changed language on the web site needs to make it clear that participation in IPMA events is not restricted but organization membership is. The membership database should be corrected by deleting the records of the people who do not qualify.
  
  
• By-Law Article VII, section 1, states "The Board of Directors will hold regular monthly meetings. The schedule for regular monthly meetings will be announced at the regular January meeting of the Board of Directors." is being replaced with "Notice of regular, special, and annual meetings of the Board of Directors will be sent at least five (5) business days prior to the day such meeting is to be held." Old Article VII, sections 1 and 3 are deleted.

Committee Reports

Budget/Finance:

• Phil Coates reported that in 2003 $18,000 and $25,000 endowed scholarships to South Puget Sound Community College and Evergreen State College respectively, fulfilled IPMA philanthropic commitments.
  
  
• Mike McVicker and Christy Ridout will complete the 2002 and 2003 IPMA Annual Reports as required in the By-Laws. Each committee was asked to submit their annual reports to Christy by February 13 so the 2003 annual report can be compiled.
  
  
• Phil Coates proposed a change to the accounting of Corporate Sponsorship (CS) payments. The CS payments would be allocated to the IPMA administrative account when received. After all CS payments have been received a credit would be made to the Forum cost center  in an amount equal to the current vendor registration fee, times the number of sponsors. The Executive Seminar would be issued a credit for the amount equal to the current participant registration fee, times the number of sponsors. The proposal was rejected. The issue was that it appeared to be a budget reduction for both the Forum and the Executive Seminar.

Communications:

• Should Newsletter articles from vendors be permitted? Mary Ellen Bradley posed the question to the board because there had been an inquiry. The board determined that it didn’t have a problem if the vendors were Corporate Sponsors and/or Forum Vendors. Mary Ellen was directed to pursue the issue with Bob Monn, the IPMA web master.
  
  
• Mary Ellen led a discussion of a proposal to require two articles a year from each board member. It was decided that given the possibility of adding articles from vendors, the requirement should remain at one article per year per board member. Mary Ellen will develop a new schedule for 2004 and present it at the March board meeting.
  
  
• Mary Ellen was directed to explore possible ways that the board might express its appreciation to Bob Monn for his continued efforts as IPMA web master.

Professional Development: Dennis Jones and Sheryl Hall reported on the lessons learned in 2003 and the committee plans for 2004.

• The committee tested three new concepts in 2003:
  
  
  • Afternoon sessions were held on several occasions to attract an audience that might have trouble making morning sessions. It was concluded that we were more likely to lose audience members late in the day and that there was no apparent advantage to afternoon sessions. The 2004 seminars will be morning sessions.
    
    
  • Some Tuesday and Thursday seminars were provided to accommodate those that have difficulty with Wednesday sessions. This seemed to work well so the committee will continue with a mix of Tuesday, Wednesday and Thursday seminars.
    
    
  • A hosted lunch seminars was held last August. It was very well received and that format will be repeated in August 2004.
  
  
• The committee plans to have six seminars in the current half-day format in 2004. The topics for all but one have been determined and the presenters for three of the six have been scheduled.
  
  
• The board concurred with the committee recommendation to have a paid "featured speaker" for the August session. The decision was preceded by a discussion about how much should be paid for this. Also, Sheryl and Dennis asked for help and ideas on what to include in the August session, for example, the theme, speaker names, etc.  The idea of "change survival" or a "motivation in times of unknown change" speaker had support of the board.  The committee asks that board members keep thinking about the event and pass ideas on to them.

Executive Summit 2004: Phil Grigg and Darrel Riffe reported:

• The 2004 Executive Seminar will be held at Campbell’s Resort on Lake Chelan.
  
  
  • A larger bus will be chartered to provide transportation to and from Campbell’s as it was very well utilized in 2003.
    
    
  • A dinner cruise is being planned pending arrangements.
  
  
• A firm reservation to hold the 2005 event at the Ocean Shores Shilo has been made.
  
  
• A soft reservation to hold the 2006 event at the Ocean Shores Shilo has also been made.
  
  
• Phil also noted that they wanted to check in to some additional locations/possibilities for the future such as Sun Mountain Lodge.

Forum 2004: Dennis Laine reported on the preparation for the 2004 Forum.

• Forum 2004 will have 20 seminar sessions, over the two days. They will be held in two rooms simultaneously and will be one hour sessions.
  
  
• There will also be 20 sessions (also 1 hour each) for corporate sponsors to present any topic they choose.
  
  
• The search for keynote speakers continues.
  
  
• Dennis thinks they have a speaker for the executive lunch who is the recognized inventor of "Hype Cycle Theory."
  
  
• The 2004 Forum budget has been developed and has been submitted for inclusion in the total IPMA budget.
  
  
• Other Forum planning is progressing at the normal pace.

Additional topics:

Storage space: Phil Coates reported that 5'x5'x8' storage can be obtained in the greater Olympia area from a low of $39 to a high of $94 a month. Several other alternatives were discussed but in the end the board directed Phil to visit the $39 site and see if it will meet the IPMA needs. If it will, the board will make a determination at the March board meeting.

Microphone purchase proposal: Dennis Jones reported that no progress had been made since the last board meeting. He pointed out that in light of the fact that St. Martin’s is acquiring a new and improved sound system, it might be prudent to wait until it is installed and see if the issue still exists. The board concurred with Dennis’ suggestion to wait.

IPMA philanthropic endeavors: After considerable discussion the board determined that while the establishment of endowed scholarships at South Puget Sound Community College and Evergreen State College were a good thing for the IPMA to have done in 2003, it should not become the organization's primary focus.

Most board members agreed that the scholarships were not directly related to the IPMA mission of providing support to state government information technology workers. It was agreed that philanthropic initiatives would only be considered at the end of the year, along with other alternatives, if funding is available.

Each committee (Forum, Professional Development, Executive Seminar) was asked to come to the next board meeting with some ideas of what they would do if they had additional funding available to them.

During the discussion of Philanthropy and spending alternatives, the board was led to the idea of formulating some type of management training series to help train mid level managers to become senior level IT executives. Something like the current Project Management UW course that is sponsored by DOP. Dennis Jones volunteered to working on exploring this rather than co-chair the Professional Development committee.

Board Service Recognition Plaques: Shelagh Taylor asked if she should proceed with ordering plaques for board members who have completed terms as had been the custom in the past. The board directed her to proceed.

Date of 2005 annual board retreat will be January 27 & 28, 2005.

The next monthly Board Meeting will be held at the Shipwreck Cafe on March 11, 2004.

The meeting was adjourned January 30, 2004, at 10:05 a.m.

Return to Top

IPMA, P.O. Box 1943, Olympia, WA 98507-1943