Wed, Jun
4 New Articles

Do You Need to Include Encryption Keys in Your HA/DR Plans?

Compliance / Privacy
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

Key management solutions create, store, serve, and manage the life cycle of the secret keys needed for strong encryption.


Security auditors have begun placing new focus on encryption key management. Because the loss of encryption keys can compromise the security of all encrypted information, compliance auditors want enterprise customers to use best practices in the management and protection of encryption keys. This means integrating your encryption key management system into your high availability (HA) and disaster recovery (DR) plans.


There is a definite trend for auditors to require that customers separate encryption keys from the data they protect. For IBM i customers, this means moving encryption keys to a separate logical partition (LPAR) or to a physically separate server. In addition to key separation, auditors want to ensure that encryption keys are protected by Master Key Encryption Key (MKEK), are generated using approved methods, are securely distributed, and are managed appropriately. Therefore, you must have a proven method of failing over to a high availability key management system and be able to recover your key management systems.


In this article, we'll talk about the basic components and requirements of a symmetric encryption key management solution and ways to insure good HA and DR handling.

Creating and Securely Storing Keys

The first job of an encryption key management solution is to create and store an encryption key. There are many methods used to create encryption keys, including random number generation (RNG), pseudo random number generation (PRNG), password-based encryption (PBE), and others. Most modern key management systems use RNG to create keys.


It turns out to be amazingly difficult to generate a truly random number. The consistency and predictability of computer systems works against making anything random, and this presents a challenge to security software engineers. The IBM i provides a system API for random number generation (search the IBM System i and IBM i Information Center for CEERAN0), but you must take care when using this API. It must be properly seeded, and you must understand the storage format for floating point numbers. However, with the help of the National Institute of Standards and Technology (NIST) guidelines on random number generation, you can successfully use the IBM i APIs to generate encryption keys from random numbers. NIST also provides validation tests that you can use to determine if you've got it right.


Once you create an encryption key, you have to securely store it for later use by encryption applications. As you might expect, there are special rules for storing encryption keys. You should use a Master Key Encryption Key (MKEK) to encrypt your encryption key. The MKEK cannot be stored in the same database as your data encryption keys, and the MKEK should be unique for each organization. While there are no formal rules for storing the MKEK, it is often stored in a hardware security module (HSM).


In addition to protecting your data encryption keys with a master key, you should also provide integrity checks to prevent key corruption or replacement. The most common way of implementing integrity checking is through the use of a Hash Message Authentication Code (HMAC). An HMAC is an encrypted one-way hash algorithm that is generated from the encryption key's attributes. You create the HMAC of an encryption key after you generate the key and before you store the key. The HMAC is then stored with the encryption key in the key database. When you retrieve the encryption key, you can regenerate the HMAC value and compare it to the original value. If it's the same, you know that your key is not corrupt.


One big area to pay attention to in key creation and storage is the master encryption key. In the event that you activate your HA plan, will the new server have a copy of the MKEK? If you use the services of the IBM i hardware key storage for your MKEK, you have to take some special steps to copy the MKEK to a new IBM i partition or platform. If you use a key management solution, be sure it fits into your HA failover strategy.


It's probably becoming clear that implementing key management can be a lot of work. But today's key management solutions are making this complex task easy to implement and automatic to administer.

Key Naming and Versioning

In order to make key management and retrieval user-friendly, keys can be given common names. For example, if you create an encryption key to protect credit card numbers in your order management system, you might name the key ORDERS. This name is then used to retrieve the key to protect a credit card number.


Over time, you may decide you want to change the key. Periodic changes to encryption keys is considered good security practice, and your compliance auditor will probably give you guidance on how frequently to change keys. When you change an encryption key, this new key is a new "version" or "instance" of the original key, and it becomes the key you will use for all new encryption. The older versions of the key remain in the key database until you completely revoke or delete the set. This provides an auditable trail of key versions and changes.


Pay special attention to how manual and automatic key rollover affects your HA and DR strategy. If you change an encryption key to generate a new version, how long will it take to mirror the new version? When will the new version be available for the DR system? If new versions of keys are not integrated closely with your HA and DR plans, you may suffer the loss of data that cannot be decrypted.

Securely Distributing Keys for Use with Encryption

Now that you've taken extraordinary steps to protect encryption keys, your next job is to securely give the key to the applications that need to perform encryption and decryption tasks. This is the function of the key server portion of a key manager and is usually implemented as a secure Transport Layer Security (TLS) service. An application that needs an encryption key opens a secure TLS connection to the key server, authenticates itself to the key server, and requests an encryption key. If the request is approved by the key server, the data encryption key is retrieved from the key store and returned to the application. Of course, the process of retrieving the encryption key involves making sure that the data encryption key is valid and has not suffered corruption or replacement.


The process described above is known as a "wire" protocol--that is, the encryption key is returned over the secure TLS connection and can be directly used by the application. An alternative retrieval method requires an application on the client side to process the returned key. This is known as an "API" retrieval method, and it normally involves decrypting the key package with RSA encryption. Key management standards are still being developed but seem to be favoring "wire" protocol over "API" retrieval.



Figure 1: Key retrieval requires a secure management plan.

Audit and Reporting

When encryption keys are being sent back and forth between applications and users, it's very important to keep an activity log of successful key retrieval requests as well as denied requests. These audit records let a security administrator review key use and identify potential abuses. Audit records should identify the source of a key retrieval request, the date and time, and the name of the keys requested. Security failures should be clearly noted in the log.


A key management application should implement system logging to support the enterprise log management and alerting system. For example, if a customer has deployed a Security Information Management (SIM) solution such as Solutionary, LogRhythm, Alert Logic, or ArcSight, the key manager should send security events to the SIM system for monitoring and alerting capabilities. This is required for the proper monitoring of key server access.


Audit trails are very important in a disaster recovery event. They help you understand the state of your system when you recover at a DR site. A good key management system will have audit trails on all key management activity prior to the DR event.

Access Controls

A key manager should enforce key retrieval access controls at a number of levels to ensure against the loss of encryption keys. These access controls usually involve TLS certificate validation, validation of the originating IP address, authentication of the user and the user's role, group membership verification, and authorization for the use of the encryption key.


In addition to key retrieval, you must have good controls over administrative access to the encryption key management functions. Two-factor authentication of key administrators is good practice, and every access to the key management system should be recorded in an audit trail and reviewed periodically.


Many IBM i key management solutions integrate user profiles into access control lists and logging. Be sure that your HA/DR strategy is integrated with your management of user profiles. You don't want to have authority errors due to missing user profiles.

Managing the Encryption Key Life Cycle

Encryption keys have a life cycle. They are created, then used, then retired from use, and finally archived or destroyed. The encryption key life cycle is well defined by NIST, and most key management systems implement life cycle management of the keys. These are the basic milestones in the life of a key:


1. Key is created.

2. Key is available for use.

3. Key is available for decryption only (no longer used for encryption).

4. Key is revoked and no longer available for use.

5. Key is escrowed.

6. Key is destroyed.



Figure 2: Encryption keys are created, used, and then destroyed.


Key management systems let you manage an encryption key through its entire life cycle. For example, when you create a key, you can specify immediate activation or choose a future activation date. The key manager will not allow the key to be used before the activation date you set.


At a certain point, you may need to retire the encryption key and no longer allow it to be used for encryption. A key manager will let you specify a future expiration date of a key at the time it's created, or the key can be manually retired at any time during its life cycle. Key retirement is generally based on the data encryption policies of your organization or those defined by compliance regulations.


It's worth noting that at any point in an encryption key's life (before it's destroyed), the key may be compromised. This is a special case, and it is defined as the loss or suspected loss of the encryption key.


At the end of an encryption key's life cycle, you may wish to destroy it to prevent its further use. For example, if a tape is lost, you can destroy the key to effectively destroy the tape. For security reasons, key destruction is an irreversible process. Once a key is destroyed, it cannot be recovered.


Because the changes to a key's status directly affect the use and availability of the key, your key management solution should dynamically mirror status changes to an HA server. In relation to your DR plan, you should back up your key management key store for your DR site any time there are significant changes to your encryption keys.


Important: Never back up encryption keys to the same archival tape as production data, and never send encryption key backups to an off-site repository in the same physical shipment with production data.

Key Rollover (or Key Rotation)

Many compliance regulations require that you change encryption keys at a regular interval. This changing of encryption keys is known as "key rollover" or "key rotation." To help you meet this requirement, good key management solutions implement both manual and automatic key rollover capabilities. For automatic key rollover, you define the number of days between rollovers. For example, you might want a key to roll over every 90 days. Once defined in your key management solution, the key change will be performed automatically.


To help you keep track of these key changes, most key managers define versions (or instances) of a key. For example, when you create a key, you give it a name, such as ORDERS. When the key is created, it gets the first version of the key. When you change the key, it gets a new version, which becomes the default version of the key. An encryption key may have many versions over time.


The newest instance of an encryption key is typically the default version. That is, when you retrieve an encryption key, you would normally ask for the current (default) version. For decrypting data, you can request a specific version of the key that matches the date of the encrypted data.


Key rotation has the same impact on your HA and DR strategy as the change to any key attribute. Be sure to mirror and back up your keys in the same way you manage your production systems.

Key Import and Export

As data encryption permeates storage and transmission of data, you will soon run into the challenge of sharing encryption keys with your vendors and customers. A key management solution should provide a method of importing encryption keys from another party and exporting encryption keys to another party. You may even find it necessary to import and export keys for applications within your own network.


There is currently no standard for how keys are imported and exported. We've encountered a wide variety of techniques and find that most key management solutions support the import of raw keys or Base64 encoded keys. Be sure that any key manager solution you use includes support for this basic type of key import and export.

Mirroring for Disaster Recovery and High Availability

As you can see, key management solutions are an important part of your IT infrastructure and will quickly become a part of your business continuity plan. Key management solutions must support high availability mirroring to accommodate HA failover and disaster recovery. This is especially important if the key manager is providing automatic key rollover. You will always want new encryption keys to be available in the event of a system failure.


One important part of the mirroring implementation is the mirroring of the MKEK. Since this key is separate from the data encryption keys, be sure you properly mirror it to the HA platform. This can be a challenge for encryption keys stored in hardware, but it must be addressed for an HA implementation.

Protect Your Keys

Key management solutions are specialized applications designed to protect the most sensitive part of the encryption process. These systems create, store, serve, and manage the life cycle of the secret keys needed for strong encryption. Understanding the importance of key management and the basics of how it works should help you to choose and implement a symmetric key management solution for your organization.


Patrick Townsend

Patrick Townsend is president of Patrick Townsend Security Solutions. The company provides encryption and key management solutions for IBM System i, IBM System z, Windows, Linux, and UNIX platforms.



Support MC Press Online

$0.00 Raised:

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: