Carol discusses the actions to review that may indicate your system is under attack or has been breached.

Last month, I discussed the unthinkable—that your IBM i system has been breached. This month, I want to discuss ways that you can look for signs of a breach or an attack, as well as the information you'll want to retain to aid the investigation should a breach occur. Both require that your system be configured properly prior to the breach. Let's take a look.

IBM i Audit Journal

Everyone knows that I'm a huge advocate of using the audit journal; there's a wealth of information that can be used to make security administrators' daily lives easier. But with the right auditing configured, there's also activity that you can look for to detect a breach or other suspicious behavior. Information in the audit journal is also extremely useful if a breach has occurred so that a forensic investigation can be performed.

The easiest way to ensure you have everything you need is to turn on all possible system-wide security actions—that is, to specify all types of action auditing in the QAUDLVL system value: *AUTFAIL, *CREATE, *DELETE, *JOBDTA, *NETCMN, *OBJMGT, *OPTICAL, *PGMADP, *PRTDTA, *SAVRST, *SECURITY, *SERVICE, *SPLFDTA, and *SYSMGT. However, while doing so is a great benefit to an investigation, you'll first want to understand the impact to your system. Both *JOBDTA, which logs the activity of all jobs (start, stop, etc.), and *SPLFDTA, which logs the activity of all spooled files (print, delete, etc.), produce huge numbers of audit journal entries. While this is valuable information, you'll have to balance the increase in audit journal entries with the increased consumption of DASD and potentially the need to save and remove audit journal receivers more often. If you need to be more selective about what's audited, one method that you can use is to enable most action auditing at the system level and additional action auditing at the user level. For example, I encourage you to consider enabling additional auditing on the IBM-supplied profiles of QSECOFR, QUSER, QPGMR, QSYSOPR, QSRV, and QSRVBAS. Why these profiles? These profiles are shipped with every system and are well-publicized on the hacker community boards. If hackers don't know anything else, they know these profiles are on the system. Additional candidates for additional auditing include all profiles with *ALLOBJ assigned directly to their profile or to one of their groups. Another level of auditing to consider is available only at the user level—that's *CMD auditing. Profiles with *CMD auditing enabled have all CL commands logged when run from either a command line or a CL program. I recommend that *CMD auditing be enabled for every profile that has access to a command line (that is, is limited capability *NO or *PARTIAL) and can be used for sign on (profiles cannot be used for sign on when their initial program is *NONE and their initial menu is *SIGNOFF). Use the Change User Auditing (CHGUSRAUD) command to enable auditing at the individual profile level.

Abnormal Activity

If a hacker or person attempting to do harm gets onto the system and the activity looks "normal," then it's virtually impossible to detect. However, most attacks—at some point—have abnormal activity associated with them. So what you're trying to accomplish is the detection of abnormal activity. Examples include:

• Abnormal amounts of invalid sign-on attempts or invalid sign-on attempts at strange hours of the day

• Invalid attempts or actual sign-on for specific profiles, such as the IBM-supplied profiles

• Any password changed from password of *NONE to a valid password

• Password changes for specific profiles (such as the IBM-supplied profiles); you should know when these occur

• Authority failures to files containing private or confidential information

• Creation of profiles by users other than the team expected to create them

• Password changes by users other than the users themselves or the help desk

• If you have journaling turned on for your database files, you can look for changes to critical fields that are made by users who don't normally make those changes. This will be especially helpful if most users on the system have authority to change the data. In other words, the files aren't locked down by object-level security.

• If you have exit point software in place and are only logging—rather than preventing—access, you can look for users downloading or uploading files containing private or confidential data or activity during hours when activity does not normally occur.

If your system is like most systems, there's so much activity that there's no way you'd be able to analyze the activity across the entire system, so you're going to want to think about what data would be valuable to a hacker or someone wanting to do harm to your organization and focus on detecting activity that would show someone attempting to download, change, or delete that data or generally do harm to the system itself.

While we're on the topic of examining activity, some organizations use log aggregators commonly known as Security Information and Event Management (SIEM) products. These servers aggregate log (audit) information from a multiple origins: Windows servers, routers, firewalls, application logs, etc. The idea is to spot inappropriate activity throughout your organization. Log aggregators exist that access IBM i information. Regardless of whether you're using SIEM technology or examining only the information from the IBM i, you'll want to look for indications of abnormal behavior.

Additional Information

Obviously, the more information available to perform forensics, the better. With that in mind, you may want to change your clean-up processes. Rather than clear the history log, outqs, and joblogs prior to performing your system save, consider performing the save first and then perform clean-up. That way you have as much information as possible available for a given date.

Now is also the time to look at how you're saving your audit journal receivers. Many people don't save the receivers at all, or they save whatever receivers are still on the system when their system save is performed, or they save all receivers but they're over-written in a few weeks when their tapes are rotated. Information in both audit journal and database journal receivers is critical to the analysis of a breach or other inappropriate activity. I encourage you to examine your save strategy for this object type to ensure that you save all journal receivers and keep them where they're easily retrieved for at least a year.

It's About Time

It's imperative that you coordinate all of your organization's servers with a time server to ensure all logs' timestamps are coordinated. Imagine trying to re-create a breach scenario but timestamps of one server is 30 seconds off. In computing time, that's a lifetime of activity. Having your servers' time be consistent across your organization may be the difference between being able to re-construct the activities of a breach or not. The protocol used for this time coordination is Network Time Protocol. On IBM i, this means specifying the IP address or name of a time server by using the Change SNTP Attributes (CHGSNTPA) command. The help for this command provides additional information on time servers.

Denial of Service

In V5R4, IBM provided us with the ability to detect when the system is under a denial-of-service (DoS) attack. The system has long been fending off attacks at the IP stack level—malformed packets, syn attacks, etc. —but until recently we never had a way to be told the system was under this type of attack. In V6R1, the user interface was significantly enhanced so that now you configure what is being monitored through IBM i Navigator. Open iNavigator for the system being configured, open the Security topic, and click on Intrusion Detection. For full instructions, search for the Intrusion Detection topic in the IBM i Information Center or download the PDF of the topic.

It's About Detecting Abnormal Behavior

I hope this issue of Security Patrol has made you think about the information you're gathering and the information you're examining. You should be looking for ways to detect behavior that's abnormal for your organization. Depending on your organization's requirements, this may mean that there are other pieces of information that you need to examine and/or save regularly—perhaps additional information beyond what is mentioned here. My sincere desire is that you never have to use this information but hopefully you'll be prepared if you do.