We were only just beginning to get our heads around Hadoop and Big Data in general when we find everyone is starting to talk about Big Data 2.0 – and it’s bigger, faster, and cleverer!
Hadoop, as I’m sure you know, is an open source project, and it’s available from companies like IBM, Hortonworks, Cloudera, and MapR. It provides a storage and retrieval method (HDFS – Hadoop Distributed File System) that can knock the socks off older, more expensive storage options on databases using SAN or NAS. It also means that more data can be stored. And that means not just human-keyed data, but data from the information of things (point of sales machines, sensors, cameras, etc) as well as social media. It’s an OCD sufferer’s dream come true. No need to delete (throw away) anything. But with all the data, it becomes important to find some way to ‘mine’ it – to derive information from the data that can be commercially useful. And that’s what’s happening, deeper and richer sets of results are being derived from the data that are beneficial to organizations.
With Version 2 of Hadoop, everything is faster. Data is processed at amazing speeds in-memory. The analysis is taking place at speed on terabytes of data. It also allows decisions to be made at speeds unavailable to humans. Research shows that algorithms with as many six variables out-perform human experts in most situations. This was tested on experts predicting the price of wine in future years and stock marketeers. So now, Big Data 2.0 means better decisions can be made at incredible speed.
It’s also possible for machines to learn using these techniques – such as the Google classic of having software that can identify the presence of a cat in video footage and no-one being quite sure how it is doing it.
For mainframe sites, Hadoop isn’t just some distant dream. You don’t need a room full of Linux servers to make it work – in fact that’s the clue to the solution. Much of this works very nicely on Linux on System z (or zLinux as many people still think of it). And once the data is on a mainframe, it becomes very easy to copy parts of it to a z/OS partition for more work to be done on the data. Cognos BI runs on the zLinux partition, so the first level of information extraction can be performed using that Business Intelligence tool. Software vendors are coming to market with products that run on the mainframe. BMC has extended its Control-M automated mainframe job scheduler with Control-M for Hadoop. Syncsort has Hadoop Connectivity. Compuware has extended its Application Performance Management (APM) software with Compuware APM for Big Data. And Informatica PowerExchange for Hadoop provides connectivity to the Hadoop Distributed File System (HDFS).
So what’s it like on the ground and away from the PowerPoint slides? At the moment, my experience is that really big companies – Google, Amazon, Facebook, and similar are pushing the envelope with Big Data. But it seems that many large organizations aren’t strongly embracing the new technology. Do banks, insurance companies, and airlines – the main users of mainframes – see a need for Big Data? Seemingly not – or not yet. Perhaps they are waiting for money to be spent and mistakes to be made before they adopt best practice and reap the benefits. Perhaps they are waiting for Big Data V3?
Big Data is definitely here to stay and those companies that could benefit from its adoption will gain a huge commercial advantage when they do.
Showing posts with label Informatica. Show all posts
Showing posts with label Informatica. Show all posts
Saturday, 1 March 2014
Sunday, 10 November 2013
Guide Share Europe 2013
The Guide Share Europe Conference at Whittlebury Manor was as excellent this year as in previous years. I was only able to make Day 1 on 5 November, but I had a great day. Apart from the 5 star presentations, it’s always fun to catch up with old friends and people I’ve spoken to at webinars, but never actually seen. Plus there’s an opportunity to catch up with a number of vendors and find out what’s happening with them and business in general.
The day started with a couple of keynotes from Tesco’s Tomas Kadlec talking about Technology the retailers battlefield – zSeries reports for duty!”, and the University of Bedfordshire’s Dr Herbert Daly talking about “Re-framing the mainframe: new generations and regenerations on System z”.
I chair the Virtual IMS user group and the Virtual CICS user group, so my time is always split between the CICS and IMS sessions. This year, I started with IBM’s Kyle Milner’s “z/OS Explorer and CICS Explorer (5.1.1)”. Kyle started by saying that z/OS Explorer is really called IBM Explorer for z/OS, and V2.1 is separate from CICS Explorer, which integrates with it. Z/OS Explorer is a desktop tool that integrates with MQ Explore, Data Studio, Rational tools (RTC), Rational Developer for z (RDz), IMS Enterprise Suite Explorer, and other tools. It allows users to view, delete, and create files on z/OS and Linux; work on SPOOL files; and much more. It’s installed using Installation Manager in a process that’s much like getting apps from the apps store. CICS Explorer lets users create new program definitions, clone resources, and other life-cycle operations. It’s built on the Eclipse framework V4.2.2.
IBM’s Greg Vance spoke about “GDPS Active-Active and IMS replication”. He described how recovery involved two concepts: the Recovery Point Objective and the Recovery Time Objective. He looked at how recovery had evolved to the point where people wanted almost immediate recovery with almost no lost data. Active-Active replication involves stopping sending transactions to one database, waiting for the last transaction to replicate across, and then sending transactions to the second database. The transactions caught before the switch over just appear a bit slow to the user. For this to work with IMS, you need InfoSphere Data Replication for IMS for z/OS V1.11. Because it uses asynchronous replication, there are no restrictions on the distance between databases. There’s low latency because of the use of parallelism. And there’s transaction consistency.
Back at the CICS stream, I saw IBM’s David Harris talk about “Eliminating the batch window with modern batch”. David starting by explaining that batch jobs often had to run when online services were down because they needed exclusive access to resources. However, there are many drivers to keep the online system available the whole time leaving little or no time for traditional batch applications to run. The solution he proposed involved running batch at the same time as the online system – and this batch system relied on Java. It comes with a big plus in that it can run in a zAAP coprocessor. Using the Batch Data Stream Framework (BDSF), it allows checkpointing, and business objects can be re-used. The batch container is a long-running CICS job, and WebSphere Application Server (WAS) is used to schedule the jobs into CICS.
Informatica’s John Boyle spoke about “IMS test data management”. He explained that we needed only a subset of data to work on in testing, and we need to hide sensitive data – particularly in light of data privacy legislation and to reduce the risk of sensitive data loss. John also stressed the need for the test data to be kept current and internally consistent. Data masking is the technique that hides personal data and it must be consistent across the data. Test software must allow policies to be applied and maintain referential integrity. It first has to establish what to mask and how to mask.
IBM’s Paul Fletcher spoke about “IMS 13 Native SQL for COBOL”. A pre-req for this COBOL V5.1, which has only just been released. COBOL programs supply SQL keywords and support static and dynamic SQL. At the moment, IMS supports only dynamic SQL. Users need to declare tables, define an SQL Communication Area (SQLCA), which is like a PCB, possibly define an SQL Descriptor Area (SQLDA), declare data items for passing data between IMS and host languages, code SQL statements to access IMS, check the SQLCA to verify the execution of the SQL statements, and handle any SQL error codes. Paul told the group that there are three types of dynamic SQL. Firstly, where the whole SQL is known when the program is written. Secondly, where the SQL is known but the value can vary. And, thirdly, where none of the SQL is known – it’s read from a file. Users need to fully qualify all tables and columns; use a WHERE clause for key fields; and use PREPARE.
The exhibition hall was packed and lively, giving people a chance to find out about various products and services. An excellent day of learning and networking was rounded off by fireworks and a barbecue dinner. I’m sorry I couldn’t make the second day. If you didn’t make GSE this year, I recommend that you go next year.
The day started with a couple of keynotes from Tesco’s Tomas Kadlec talking about Technology the retailers battlefield – zSeries reports for duty!”, and the University of Bedfordshire’s Dr Herbert Daly talking about “Re-framing the mainframe: new generations and regenerations on System z”.
I chair the Virtual IMS user group and the Virtual CICS user group, so my time is always split between the CICS and IMS sessions. This year, I started with IBM’s Kyle Milner’s “z/OS Explorer and CICS Explorer (5.1.1)”. Kyle started by saying that z/OS Explorer is really called IBM Explorer for z/OS, and V2.1 is separate from CICS Explorer, which integrates with it. Z/OS Explorer is a desktop tool that integrates with MQ Explore, Data Studio, Rational tools (RTC), Rational Developer for z (RDz), IMS Enterprise Suite Explorer, and other tools. It allows users to view, delete, and create files on z/OS and Linux; work on SPOOL files; and much more. It’s installed using Installation Manager in a process that’s much like getting apps from the apps store. CICS Explorer lets users create new program definitions, clone resources, and other life-cycle operations. It’s built on the Eclipse framework V4.2.2.
IBM’s Greg Vance spoke about “GDPS Active-Active and IMS replication”. He described how recovery involved two concepts: the Recovery Point Objective and the Recovery Time Objective. He looked at how recovery had evolved to the point where people wanted almost immediate recovery with almost no lost data. Active-Active replication involves stopping sending transactions to one database, waiting for the last transaction to replicate across, and then sending transactions to the second database. The transactions caught before the switch over just appear a bit slow to the user. For this to work with IMS, you need InfoSphere Data Replication for IMS for z/OS V1.11. Because it uses asynchronous replication, there are no restrictions on the distance between databases. There’s low latency because of the use of parallelism. And there’s transaction consistency.
Back at the CICS stream, I saw IBM’s David Harris talk about “Eliminating the batch window with modern batch”. David starting by explaining that batch jobs often had to run when online services were down because they needed exclusive access to resources. However, there are many drivers to keep the online system available the whole time leaving little or no time for traditional batch applications to run. The solution he proposed involved running batch at the same time as the online system – and this batch system relied on Java. It comes with a big plus in that it can run in a zAAP coprocessor. Using the Batch Data Stream Framework (BDSF), it allows checkpointing, and business objects can be re-used. The batch container is a long-running CICS job, and WebSphere Application Server (WAS) is used to schedule the jobs into CICS.
Informatica’s John Boyle spoke about “IMS test data management”. He explained that we needed only a subset of data to work on in testing, and we need to hide sensitive data – particularly in light of data privacy legislation and to reduce the risk of sensitive data loss. John also stressed the need for the test data to be kept current and internally consistent. Data masking is the technique that hides personal data and it must be consistent across the data. Test software must allow policies to be applied and maintain referential integrity. It first has to establish what to mask and how to mask.
IBM’s Paul Fletcher spoke about “IMS 13 Native SQL for COBOL”. A pre-req for this COBOL V5.1, which has only just been released. COBOL programs supply SQL keywords and support static and dynamic SQL. At the moment, IMS supports only dynamic SQL. Users need to declare tables, define an SQL Communication Area (SQLCA), which is like a PCB, possibly define an SQL Descriptor Area (SQLDA), declare data items for passing data between IMS and host languages, code SQL statements to access IMS, check the SQLCA to verify the execution of the SQL statements, and handle any SQL error codes. Paul told the group that there are three types of dynamic SQL. Firstly, where the whole SQL is known when the program is written. Secondly, where the SQL is known but the value can vary. And, thirdly, where none of the SQL is known – it’s read from a file. Users need to fully qualify all tables and columns; use a WHERE clause for key fields; and use PREPARE.
The exhibition hall was packed and lively, giving people a chance to find out about various products and services. An excellent day of learning and networking was rounded off by fireworks and a barbecue dinner. I’m sorry I couldn’t make the second day. If you didn’t make GSE this year, I recommend that you go next year.
Sunday, 3 November 2013
When worlds collide
We know that mainframes are rock solid workhorses that ensure the banks and insurance companies and airlines and pretty much every other large organization get their work done correctly and swiftly. And we know that access to mainframes has been extended outside the world of green screens to anyone on a browser with proper authorization. And we also know that there’s little distinction between the world of cloud computing and distributed mainframe computing. But the latest big thing is Big Data – and that seems like a different world.
Big Data is used to refer to huge amounts (exabytes) of data, often unstructured, that can originate from a variety of sources – such as cameras, weather satellites, credit card machines, barcode readers, the Internet of Things, anything! This Big Data usually sits on Linux or Windows boxes and some of the early developers were Google, Amazon, and Facebook. The data is stored in HBase, a non-relational, distributed database, written in Java. And the file system is what’s called a Hadoop Distributed File System (HDFS). At runtime, a process maps the data and reduces it – that’s called MapReduce.
So how do these two worlds come together? For a start a lot of the things you need for Big Data are Open Source and come from the Apache Foundation. IBM is a member of the foundation and has a number of products that extend Big Data’s functionality. IBM provides InfoSphere BigInsights, Data Stage, Streams, and Guardium. There’s Big SQL with Big Insights V2.1, and the spreadsheet-like Big Sheets.
If you want to run Big Data – Hadoop – on your mainframe, you’ll need to do it in a Linux partition (Linux on System z). But IBM isn’t the only mainframe software vendor that’s getting in on the act. We’ve recently heard from BMC, Syncsort, Compuware, and Informatica about their products.
BMC has extended its Control-M automated mainframe job scheduler with Control-M for Hadoop. The product enables the creation and management of Hadoop workflows in an automated environment and is aimed at Hadoop application developers and enterprise IT administrators who are using Hadoop as part of their production workload.
Syncsort has Hadoop Connectivity, which prevents Hadoop becoming another silo within an enterprise. The product makes it easy to get data in and out of Hadoop. The product provides: native connectivity to all major data sources and targets; native mainframe connectivity and support for EBCDIC/ASCII, VSAM, Packed decimal, Comp-3, and more; heterogeneous database access on Hadoop; direct I/O access for faster data transfers; and high-performance compression.
Compuware has extended its Application Performance Management (APM) software with Compuware APM for Big Data. This, they claim, allows organizations to tame Big Data applications to eliminate inefficiencies and rapidly identify and resolve problems. Using PurePath Technology, it provides visibility into Hadoop and NoSQL applications. Organizations, they say, use Compuware APM for Big Data to reduce costs, analyse issues, and ensure optimal efficiency from their Big Data investments.
Informatica PowerExchange for Hadoop provides native high-performance connectivity to the Hadoop Distributed File System (HDFS). It enables organizations to take advantage of Hadoop’s storage and processing power using their existing IT infrastructure and resources. PowerExchange for Hadoop can bring any and all enterprise data into Hadoop for data integration and processing. Fully integrated with Informatica PowerCenter, it moves data into and out of Hadoop in batch or real time using universal connectivity to all data, including mainframe, databases, and applications, both on-premises and in the cloud. Informatica PowerCenter Big Data Edition is, they claim, highly scalable, high-performance enterprise data integration software that works with both Hadoop and traditional data management infrastructures.
Clearly, these two different worlds have more than collided – we are beginning to see the integration of these previously quite separate worlds with software from a number of vendors helping users with the integration process. And as users, we get the best of both worlds!
Big Data is used to refer to huge amounts (exabytes) of data, often unstructured, that can originate from a variety of sources – such as cameras, weather satellites, credit card machines, barcode readers, the Internet of Things, anything! This Big Data usually sits on Linux or Windows boxes and some of the early developers were Google, Amazon, and Facebook. The data is stored in HBase, a non-relational, distributed database, written in Java. And the file system is what’s called a Hadoop Distributed File System (HDFS). At runtime, a process maps the data and reduces it – that’s called MapReduce.
So how do these two worlds come together? For a start a lot of the things you need for Big Data are Open Source and come from the Apache Foundation. IBM is a member of the foundation and has a number of products that extend Big Data’s functionality. IBM provides InfoSphere BigInsights, Data Stage, Streams, and Guardium. There’s Big SQL with Big Insights V2.1, and the spreadsheet-like Big Sheets.
If you want to run Big Data – Hadoop – on your mainframe, you’ll need to do it in a Linux partition (Linux on System z). But IBM isn’t the only mainframe software vendor that’s getting in on the act. We’ve recently heard from BMC, Syncsort, Compuware, and Informatica about their products.
BMC has extended its Control-M automated mainframe job scheduler with Control-M for Hadoop. The product enables the creation and management of Hadoop workflows in an automated environment and is aimed at Hadoop application developers and enterprise IT administrators who are using Hadoop as part of their production workload.
Syncsort has Hadoop Connectivity, which prevents Hadoop becoming another silo within an enterprise. The product makes it easy to get data in and out of Hadoop. The product provides: native connectivity to all major data sources and targets; native mainframe connectivity and support for EBCDIC/ASCII, VSAM, Packed decimal, Comp-3, and more; heterogeneous database access on Hadoop; direct I/O access for faster data transfers; and high-performance compression.
Compuware has extended its Application Performance Management (APM) software with Compuware APM for Big Data. This, they claim, allows organizations to tame Big Data applications to eliminate inefficiencies and rapidly identify and resolve problems. Using PurePath Technology, it provides visibility into Hadoop and NoSQL applications. Organizations, they say, use Compuware APM for Big Data to reduce costs, analyse issues, and ensure optimal efficiency from their Big Data investments.
Informatica PowerExchange for Hadoop provides native high-performance connectivity to the Hadoop Distributed File System (HDFS). It enables organizations to take advantage of Hadoop’s storage and processing power using their existing IT infrastructure and resources. PowerExchange for Hadoop can bring any and all enterprise data into Hadoop for data integration and processing. Fully integrated with Informatica PowerCenter, it moves data into and out of Hadoop in batch or real time using universal connectivity to all data, including mainframe, databases, and applications, both on-premises and in the cloud. Informatica PowerCenter Big Data Edition is, they claim, highly scalable, high-performance enterprise data integration software that works with both Hadoop and traditional data management infrastructures.
Clearly, these two different worlds have more than collided – we are beginning to see the integration of these previously quite separate worlds with software from a number of vendors helping users with the integration process. And as users, we get the best of both worlds!
Subscribe to:
Posts (Atom)