Continuous availability – no longer a dream?

Zero downtime is a goal that many companies are striving for. It sounds so straighforward, and yet it’s not that simple to achieve – especially when it involves the continuous availability of large, high-volume databases. One of the inherent problems is that data replication for high-availability is filled with many nuances that need to be addressed for a successful deployment, including maintaining sub-second latency, active/active considerations, scalability options, conflict detection/resolution, recovery, exception processing, and verifying that the source/target are synchronized properly.

One of the problems that organizations face is the need to address lots of different business issues using, what often involves, multiple software packages. Integrating these different pieces of software – perhaps even from different vendors – can add an extra level of complexity to the job in hand. What those organizations really need is a single piece of software that’s flexible enough to provide a comprehensive solution for changed data capture, replication, enhancing existing ETL (Extract, Transform, and Load) processes, and data migrations/conversions. Quite a big ask.

Wouldn’t you be interested in software that offers industrial-strength, near-real-time data integration solutions that include high-performance Changed Data Capture (CDC), data replication, data synchronization, enhanced ETL and business event publishing? And what if it was equally simple to experience the high-speed delivery of mainframe data (IMS, DB2, VSAM, etc) into data warehouses and downstream applications? Too good to be true?

If you’re like me, you carry around a list of capabilities in your head, and tick them off – or more often don’t tick them off – when you give software the once over. So here’s the kind of things I’d have on my list for an integration engine. In general I’d expect:

• Concurrent operation across multiple operating system platforms
• Multi-step processes within a single script (UNION)
• Simultaneous multi-record type file handling
• Multi-level array handling (repeating groups) of source data store records/rows
• Data filtering and cleansing
• Dynamic look-up table processing
• Support for data transfer and communication using TCP/IP and MQSeries
• Preservation of referential integrity (RI) rules on target updates
• Joins/Merges of heterogeneous databases/files.

In terms of data transformation I’d like to see:

• Case (If/Else) logic
• Extensive date cleansing and formatting
• Arithmetic functions (add, subtract, multiply, etc)
• Aggregation functions (sum, min, max, avg, etc)
• Data type conversions
• String functions
• Data filtering
• XML data formatting
• Delimited data formatting.

When it comes to datastore processing I’d want:

• High performance bulk data transfer
• Concurrent processing of multiple data store types
• Creation of target data stores from source data store format
• Insert/append to existing target data stores
• Update/replace existing target data stores
• Delete from existing target data stores
• New column/field creation Data Movement.

And for Data Movement, my list includes MQSeries, TCP/IP, and FTP.

If there was also some kind of Integration Center that had an easy-to-use Graphical User Interface (GUI) enabling users to quickly develop data integration interfaces from a single control point – that would be good. Additionally, some way to develop, deploy and maintain data interfaces, create relational DDL (Data Definition Language), XML (Extensible Mark-up Language ) and C/C++ structures from COBOL Copybooks, monitor the status of integration engines, and contain an integrated metadata repository – that would be a real plus.

I’d definitely want to find out more about a single piece of software that provided high-performance Changed Data Capture (CDC) and Apply, data replication, event publishing, Extract, Transformation, and Load (ETL), and data conversions/migrations.

So, if you’re like me and want to know more, there’s a webinar from SQData’s Scott Quillicy on 1 December at 2pm GMT (8am CST). To join the webinar from your PC, you need to register before the event at https://www1.gotomeeting.com/register/844029904. I’ll see you there.

The importance of mainframe performance

It’s so easy to forget, or just take it as read, that mainframes have been able to successfully run with five nines availability for well over a decade. What that means is achieving 99.999% of scheduled uptime. In other words, it means that unscheduled downtime is less than five and half minutes in a year! Now that kind of amazing performance is something that boxes running other operating systems can only dream of. Some are working towards that level of availability, but others (you know who I’m thinking of here) aren’t even close.

But I wasn’t thinking about performance in that sense. We just take it for granted the operating system is going to be always working. What I was thinking about was the performance of the major subsystems running under z/OS. It’s very important to take steps to ensure that CICS or IMS are performing optimally. Monitoring software can be installed that will identify when preset thresholds are reached. They help identify bottlenecks and then the appropriate action can be taken to resolve them. This is the kind of stuff systems programmers have been working away at for years. They’ve been using faster processors, faster I/O, more efficiently-coded transactions, until every component is working as well as it can.

In previous blogs, we’ve talked about monitoring software that can arrange for alerts to be sent to designated staff as text messages or e-mails – allowing them to access the nearest iPad or laptop and take steps to resolve the new problem.

As well as CICS and IMS, there are monitors for DB2, WebSphere, and z/OS itself. These can all be integrated and produce wonderful moving graphs or other displays that allow users to tell at a glance whether everything is OK or whether a slight tweak to the subsystem is required. In addition, we’ve had software that learns how to maintain high performance and make appropriate changes on-the-fly, without any human intervention.

But the problem that many sites now face is what they can do if, for example, IMS users are reporting slow response times, but the problem appears to be coming from outside the IMS subsystem rather than from inside it. For example, what appears to be an IMS performance problem could be a CICS, DB2, WebSphere, or z/OS performance problem. The challenge facing systems programmers in this situation is to correlate performance data in IMS with activities in these other systems in order to discover the cause of the slow response time.

One new solution is Transaction Analysis Workbench, which is an IMS tool. If you’re interested in how to approach this type of situation, how to gather the necessary information from multiple subsystems, and then analyse, diagnose, and resolve the problem, you’ll be interested in the webinar from the Virtual IMS user group this week.

The Virtual IMS user group runs free webinars every other month. During the webinar, a technical expert shares their hard-won knowledge with the rest of the group. The webinars use Citrix GoToMeeting, which means you don’t have to face the hard task of convincing your company to fund your user group experience – you just sit down at your laptop and log in.

Anyone wishing to join the webinar needs to join the user group – which is also free. The next meeting is at 10:30 Central Standard Time on Tuesday 8 February. The user group’s Web site (where you can join) is at www.fundi.com/virtualims.