18
Thu, Jul
2 New Articles

When CCSID Constants Vary, Part I

RPG
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

Learn how one simple character can make a program go crazy.

 

Several years ago, I wrote an email processing program for the System i and posted the source on my Web site. Several people have downloaded the code, compiled the program, and run it without any problems…until recently. For a few months now, people have been reporting that the program crashes. The program MAIL, which worked perfectly for several years, suddenly went haywire.

 

In this series of articles, I will show you how a CCSID variation can turn a program crazy, why this can suddenly become a problem, and why perhaps you've had the problem for years but didn't see it. This can happen even when your applications are Unicode-ready.

 

The first article will explain the basis of the problem. The second article will show you how I solved the problem. Finally, the last article will show you some tricks I learned along the way.

 

This article, the first of the series, explains the problem that needs to be solved.

 

The Coded Character Set Identifier (CCSID) is something like the alphabet. Not everybody uses the same alphabet.

 

So let's look at the MAIL program problem, which I mentioned in the opening paragraph. As an example, here's what one user reported. The procedure returned the following:

 

Invalid email address : This email address is being protected from spambots. You need JavaScript enabled to view it.
Sender email (This email address is being protected from spambots. You need JavaScript enabled to view it.) is not a valid MIME address.
RPG status 00202 caused procedure MIMESENDER in program JPLTOOLS/MAILR to stop.

 

The problem appeared simultaneously with an incoming group of new users from another country. The email addresses were perfectly valid. So why did my procedure perceive them as invalid addresses? What happened with the MAIL program? It was not recompiled, so the reason had to be elsewhere.

 

My investigation revealed a clue: the CCSID of the job that makes MAIL crazy is different from the CCSID of the job that compiled the program MAIL. Don't ask me how I figured that out or how I discovered that this coincidence is in fact the primary reason for the haywire behavior. Sometimes, we are just lucky.

 

De facto, when the job that uses MAIL has the same CCSID that the compilation job has, the problem never appears.

 

Why on earth do I have to take care of this strange parameter of the job that compiled my program when I am calling this program?

What Is the Link? 

The CCSID is an old parameter. We can retrieve the CCSID parameter from many places on the IBM i (example: SBSD, JOB, FILE, FIELD).

 

Moreover, even DSPPGM shows the following:

 

Coded character set to identify. . . . . . . . : 65535

 

But CHGPGM does not allow me to modify it.


The CCSID(65535)—sometimes named CCSID(*HEXA)—means that the object is a binary object. Let me explain. When you copy a program from one machine to another, you hope to obtain on the target machine exactly the same program (object binary code) as the one on the origin machine. When it is a binary copy, you get the same object, bit to bit.

 

CCSID is the parameter that tags an object precisely for translation purposes. The CCSID explains to the system whether, during this copy, a character translation is mandatory or forbidden. On a binary object, translation is forbidden (it is the good choice). Therefore, after a copy, you get the same object (bit to bit). Since a program is a binary object, after a copy, a program continues to work.

 

The special value CCSID(65535) or CCSID(*HEXA), which is used to tag *PGM objects, is here exactly for that purpose: automatic translation of bytes is forbidden, so a copied program continues to work.

 

Nevertheless, this generally good process has a side effect: in my program, there is a literal constant, the "at" sign (@).

 

There are some places in my program (a binary object) where I have stored text. Oops. Text translation is sometimes mandatory.

 

I resume: For a program, automatic character translation by the system is forbidden. In my program, there is a hard-coded @, and the @ is a character that varies from one CCSID to another. Remember: the initial issue is that the program goes haywire when CCSID varies.

 

Therefore, the reason for the unintended behavior is well outside of the program, but the problem is in the program. If the @ was not hard-coded in my program, I would never have had an issue.

 

The problem is precisely here:

 

// * one @ is mandatory

i = %scan ('@':email);

 

What is wrong in this code? The @ is hard-coded as a literal text constant.

 

What I have just displayed is copied from the source code. It comes from a file (and a source file is one kind of DB2 file), and this file has the CCSID 297 (I am French). My job and my 5250 emulator also use CCSID(297). When I compile my program, the compiler reads the source code and transforms it into a binary object (a *MODULE).

 

At this moment, my @ character becomes X'44 '.

 

dsppfm jpltools mailr 

 

i = %scan('@':email); 

84746A889474778988955 

90E0C2315DD4DA54193DE 

 

 

What happens in the program is therefore really a search of X'44' in the email field.

 

While the job that uses MAIL is in CCSID(297), the problem is hidden. As soon as the CCSID varies, the problem appears. If the job is, for example, in CCSID(037)—the preferred CCSID of Americans—the job transmits to the program a correct email address, with an American @. In CCSID 37, the code for @ is x'7C '.

 

Therefore, the program becomes crazy:

 

Invalid email address: This email address is being protected from spambots. You need JavaScript enabled to view it.

 

But why does the problem appear only now?   

 

Since this program walks from end to end of the Internet, the problem should have immediately appeared.

 

If I had published the object program, the issue would have appeared immediately. But I published only the source code.

 

The problem is hidden because the source code comes from the Internet. The source code I just showed you comes from my IBM i, from a source file with CCSID(297). When I want to publish it, I first copy it to my PC via FTP. Automatically, FTP translates the source code (EBCDIC) that comes from my IBM i to the CCSID of my PC (ASCII). Then the source code is published on my Web site, therefore into an ASCII file, and more precisely, in CCSID(1252), which is the Windows CCSID for America and West Europe.

 

Then, a programmer gets the source code. Say it's Dan, my American friend. Dan copies the program from the Internet to a PC, and then he copies it with FTP to his IBM i source file. Dan's source file has CCSID(37). So FTP executes automatic conversion from the ASCII CCSID(1252)  to the EBCDIC CCSID(37), the CCSID of Dan's source file.

 

When Dan looks at the code, he sees no issue (the problem is hidden):

 

dsppfm jpltools mailr 

 

i = %scan('@':email); 

84746A889477778988955 

90E0C2315DDCDA54193DE 

 

But if there is no issue, why am I writing this article? Because there is truly an issue. After Dan has compiled the program, the object code scans for x'7c' in each incoming email address. This works perfectly. Well, it works perfectly until I try to use it. Remember, I'm French. In the French CCSID, the @ is coded x'44'. When I try to use the program compiled by Dan, the object code tries to scan for x'7c' in each incoming email. This never works.

 

So, while people use only the default CCSID for their machine, there is no problem. The problem is for people who develop for multiple languages. This may happen suddenly when your company incorporates or is incorporated into a world group.

 

Let's continue: When the CCSID of the job that uses MAIL is different from the job that compiled MAIL, the program becomes crazy. The exact cause is a literal constant keyed in a line of the source code. After I identified this problem, I found on the IBM Info Center these "Recommendations and guidelines for using CCSIDs." Here's one line from that document: "Avoid using characters that are not in the invariant character set for names and literals in programs." Now, I understand what that means. Go to that page; you'll see a lot of "Be aware of…"

 

This first article has described the issue. We now are sure the problem is with CCSID.

 

In my next article, I will go deeper into the CCSID, its definition, the reason why I have a problem, and the method to implement a solution in the RPG program.

as/400, os/400, iseries, system i, i5/os, ibm i, power systems, 6.1, 7.1, V7, V6R1

Jean-Paul Lamontre

Jean-Paul Lamontre has been working on IBM machines since 1976. His first was a 3/15 with 128K RAM (the biggest machine of the county). His first program was an RPG program, no more than 15 lines. It never compiled, and nobody ever understood why.

 

Currently, Jean-Paul divides has work time between two companies.

 

For Cilasoft, which offers the Cilasoft Audit and Security suite, he is the director of development. The Cilasoft suite is a cornerstore to any company's compliance process.

 

For Resolution, which offers Xcase, a database engineering suite, he is the CTO of the IBM i department. Xcase allows developers to modernize a DDS database to DDL, discover and implement implicit relationships, and manage SQL databases using an advanced GUI.

 

Jean-Paul also publishes some free tools on his personal Web site. Most popular are SQL2XLS, SPLF2PDF, and MAIL.

BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: