Wouldn't it be nice if big data access was fast and easy with minimal setup time or struggle?

Big data: This term covers a lot of ground. With no definitive dividing line based on file size, data type, number of records, or job requirements, how do you know if you're doing big data? Ask yourself, has the size or speed requirements of new analytic initiatives put serious strain on existing IT infrastructure? Most organizations could say yes because big data is everywhere. For example:

• IBM estimates that 90% of the data in the world today was created in the last two years

• 10,000 payment card transactions are made every second around the world

• In 2011, the U.S. healthcare data alone reached 150 exabytes, and that number is expected to double each year

Big data isn't driven by the tech industry; it's a result of macroeconomics. Global markets (labor, raw materials, etc.) have emerged to drive technology. Big data and analytics are responses to competitive and economic pressure; until the data is analyzed and becomes useful information, it's nothing more than just data. With the right tool, vast amounts of relational data can allow organizations to become more efficient, exploit opportunities more quickly, and more accurately predict the future conditions.

As you collect more and more information, your relational database files continue to grow. As data volumes expand, organizations are faced with a challenging issue: how to store all that information efficiently and accurately. The greatest obstacle is how to mine the data and deliver useful information while not bogging down the system.

Partition files are the solution to big data on IBM i because of:

• Improved manageability for large tables
• The ability for various data partitions to be administered independently (able to backup and restore selected data partitions instead of the entire table)
• Increased query performance through data partition elimination
• DB2 optimizer is data partition aware. Only the relevant data partitions are scanned.
• Fast online data roll-in/roll-out
• Can easily add or remove data partitions without taking the database offline
• Better optimization of storage costs
• DB2 9 allows you to optimize your overall storage costs and improve performance
• Larger table capacity
• Greater index placement flexibility

SEQUEL allows you to work with your big data efficiently and accurately, allowing users to get the information they need, how they need it, and when they need it!

With SEQUEL you can:

• Explore your data efficiently using the SQE (SQL Query Engine)
• Query data locally on the IBM i or on other systems
• Query multi-partitioned files
• Email or display live, current, dynamic data
• Create or send results as text, Excel, PDF, HTML, or other PC documents
• Display data from a green-screen or graphical interface
• View data from a browser or mobile device with SEQUEL Web Interface

How can SEQUEL easily query multi-partitioned file, files with millions, billions, and trillions of records?

SEQUEL can query multi partition tables (files) with a single SEQUEL View. Do you have a large number of records? Are you concerned about efficiency? No problem. SEQUEL uses the new SQE to help gather the selected records quickly. With table partitioning and SEQUEL using a server parameter of *LOCALSYS, SEQUEL is able to use the newer, faster SQE (SQL Query Engine). You can actually build one query that selects data over both members of a partitioned table at the same time.

Example: SEQUEL's graphical interface (SEQUEL ViewPoint) allows users to simply drag-and-drop and double-click to build the view. With SEQUEL using the *LOCALSYS database, it's simple to scan all the records from all members.

Figure 1: The database is set to *LOCALSYS.

Figure 2: Define the view.

Figure 3: The results show all records from all table partitions returned.    

With any combination of big data, SEQUEL can deliver the results you need, how and when you need them. Partition files can easily handle your big data needs on the IBM i, but you'll need a tool to mine the data, deliver useful information to the user, and run efficiently using a fast query engine like SQE. To handle your big data partition files, you'll need SEQUEL. See how SEQUEL makes your big data access fast and easy in the "Data Access in a Big Data World" white paper.

Heath Kath is a Technical Solutions Consultant specializing in business intelligence, data access and analysis, and report distribution. Heath is a Robot and IBM Certified Specialist with more than 20 years of System i experience, including 11 years at IBM working in product support, testing, software support, and training. His background includes CL and RPG programming, Query/400, Query Manager, SQL, and journaling. Heath hosts a bi-weekly webinar for business intelligence products and provides technical support for new customers.