Like magic, custom reports and critical files in various formats appear each day in user's folders, portals, print queues, and email. But if they didn’t run one day soon, would anyone know why?

Programmers are taught early in their career to simplify program maintenance by carefully documenting their work. Companies invest significant time and money developing business continuity and disaster recovery plans in the hope they can minimize risk and reduce recovery time if the enterprise is threatened. But many of these same organizations never document their reports and reporting processes. Knowledge of these practices is frequently overlooked until the environment changes and someone stops receiving a report or file they rely on to monitor and guide their daily operations. That's a bit too late.

Over the many years I've worked for a company that provides reporting and business intelligence software, I've heard a wide range of reasons for this documentation oversight. Of course, the most common response is, "We don't have time." But there are other less-obvious reasons. Reports are often developed by power users and analysts outside the information technology department. These staff members were probably never asked or trained to document their work. Database and application development managers can argue that reports that read, but don't change, the database aren't their responsibility. Programmers may not view the logic embedded in queries and report programs as "code" that will require maintenance over time. We also know there are some who believe "queries" aren't sufficiently complex enough to warrant documentation. Anyone with this belief hasn't yet been asked to redesign or debug a complex query.

Companies that haven't documented their reports and reporting process typically get by without too many problems until the few people with in-depth knowledge aren't around to keep things running. There's no reason to wait for that day.

A definition of what we consider a "report" is important because the term has evolved over time. A narrow definition that looks only at spool file output is inadequate. Here is a more helpful definition for the sake of this discussion: a job that runs on a recurring basis that queries, outputs, exports, or transfers data and is not part of a supported application system. We'll assume any reports built into a vendor or internally developed and supported application will be maintained as part of that system. Our focus here is on all the other reports and reporting processes your company has created to support daily operations.

Your company's unique reports and reporting processes may include output generated by compiled programs, IBM Query/400 queries, queries written with third-party query and reporting solutions, SQL and PHP scripts, PC file transfers, and sometimes nightly transfers to other servers and databases. Knowing if and how your organization uses each of these methods and possibly others is critical so that you can locate all the elements that make up your reporting infrastructure. So what information do you need?

Report and reporting process documentation should illuminate your company's reporting on multiple levels.

• Report Usage—Documentation that identifies what reports are being run, who runs them, when they were last run, how often they are run, and whether they are being initiated from a job-scheduling utility or other automated process. You may have numerous CL programs that initiate reports. Try to review report usage over a timeframe that reflects your company's business cycle (weekly, monthly, quarterly, etc.) as some critical reports may only be run periodically.
  
  
• Report Design—Documentation of individual reports to show tables joined, columns used, calculations, selects, sorts, summaries, output type, and output destinations (print, display, database table, IFS folder, etc.). Your ability to generate this information depends in part on your development tools and methods. Companies using IBM Query/400 should look into using the Print_Query_Definition stored procedure or the more complete IBM Query/400 Discovery Tool to generate documentation of their Query/400 object libraries, tables, columns, selection criteria, and result fields. If you use third-party solutions, ask your software vendors for advice on how to generate this information.
  
  
• Database Access—Documentation showing what reports access what tables and columns so that you can cross-reference database changes in a new application software release with the reports that access them and identify those that should be tested. Hopefully, you can output this information to a database table so that you can look at it in various ways via queries or a program documentation utility.
  
  
• Report Origin—Documentation identifying who the report developers are, when a report was created, who last changed it, and when it was last changed. Depending on your regulatory environment or company policy, you may need to enforce change-management controls over some reports. If so, you may want to investigate query and report-writing solutions that allow you to maintain an audit trail of your changes and copies of previous report definitions.
  
  
• Report Dependencies—Documentation identifying reports that require output from other jobs or initiate other processes such as exit programs, other reports, file transfers, and email messages when finished. Business users commonly work around query tool limitations by writing queries that must be run sequentially to generate temporary files required to produce a single report. Don't assume the last job in the sequence is all you need to understand.
  
  
• Report Integration—Documentation identifying internally developed reports that are executed from application menus, web pages, or portals. Even vendor-supported ERP systems may have links to custom reports that are not part of the vendor's supported release.
  
  
• Data Transfers—If applicable in your environment, documentation of the jobs that extract and transfer data from your online transaction processing (OLTP) database to a data warehouse. This topic is outside the scope of this article. We only mention it because we've received calls from customers who discovered their nightly data transfer had failed but had no documentation to help them identify the jobs they needed to test, change, or run.

Reports and reporting processes are an integral part of operations. As reporting and business intelligence (BI) software extends to more users and use of data-driven analytics gains momentum, companies should invest the time required to create and maintain thorough documentation of their custom reports and reporting processes. Operating without report and report-processing documentation is a significant and costly oversight that is easily avoided.