IBM’s approach to Big Data

IBM has taken lots of the open source Big Data technologies – like Hadoop, MapReduce, HBase – and added its own technology – like Big Sheets, DB2, DataStage – to create something hugely more powerful.

IBM’s InfoSphere BigInsights builds on open source Hadoop capabilities for enterprise class deployments. The enterprise-level capabilities can be grouped together as: visualization and exploration, development tools, advanced engines, connectors, workload optimization, and administration and security.

IBM claims the business benefits are: quicker time-to-value because of IBM’s technology and support, reduced operational risk, enhanced business knowledge with a flexible analytical platform, and it leverages and complements existing software.

In terms of administration and security, the Web console can start and stop services, run and monitor jobs (applications), explore and modify the file system, and built-in apps make it easy to do common tasks.

The connectors link to databases like DB2, Netezza, Oracle, Teradata. And there’s integration with: InfoSphere Data Stage (data collection and integration), InfoSphere Streams (real-time streams processing), InfoSphere Guardium (security and monitoring), Cognos Business Intelligence (Business Intelligence capabilities), and IBM Platform Computing (cluster/grid infrastructure and management), and more. Big SQL is coming with BigInsights V2.1. This will provide SQL access to data stored in BigInsights through JDBC/ODBC and use rich standard SQL to leverage Map/Reduce parallelism or achieve low-latency.

Advanced engines include an advanced text analytics engine that can automatically identify and understand key information in text. Text Analytics is really useful because most of the world’s data is in unstructured or semi-structured text; social media is full of discussions about products and services; internal information in organizations is locked in blobs, description fields, and sometimes even discarded. It’s been suggested that over 80% of stored information is unstructured – such as e-medical records, hospital reports, case files, police records, emergency calls, tech notes, call logs, online media, insurance claims, Twitter, Facebook, blogs, and forums.

In terms of development tools, there is an Eclipse-based development environment for building and deploying applications. There are developer tools and a set of analytic extractors for fast adoption that reduce coding and debugging time by up to 30% (IBM claims). There are also plug-ins for text analytics, MapReduce programming, Jaql development, Hive query, etc.

Visualization and exploration has Big Sheets, providing Web-based analysis and visualization for users with a familiar spreadsheet-like interface that can define and manage long-running data collection jobs.

Meanwhile, Microsoft has identified Hadoop users as a useful market to get into. Speaking recently at the Hadoop summit, Quentin Clark, corporate VP of data platforms said: “We believe Hadoop is the cornerstone of a sea change coming to all businesses”.

Microsoft is integrating Hadoop with its products and services. And, Clark says that Microsoft intends to stick to the principles of open source by contributing to the Hadoop project, rather than simply using it and adding its own stuff. Hortonworks recently announced management packs for Microsoft System Center Operations Manager and Microsoft System Center Virtual Machine Manager – both products for administering the Hortonworks Data Platform (HDP) distribution.

Apparently Microsoft is positioning itself as a big data player with a powerful set of Business Intelligence (BI) tools. Data Explorer for Excel 2013 is a self-service BI add-in allowing users to import data from a variety of sources, including Hadoop. SQL Server 2012 Parallel Data Warehouse (PDW) is a massively parallel processing data warehousing appliance designed for Hadoop integration. Microsoft is also trying to bring Hadoop into the cloud using Windows Azure.

Businesses can’t ignore Hadoop, and the fact that major software vendors are getting behind it means it’s not going to be some flash-in-the-pan idea. Certainly, I can imagine major organizations looking to get a huge business advantage by embracing the technology now – to be ahead of their competitors. Smaller organizations will probably take a few years before they see a business case for it. By then the IBM products (and Microsoft’s) will be very mature and eminently suitable.

IBM-Oracle face-off

Having upset the open source community by legal action against TurboHercules, and worried its own customers and made life very interesting for third-party software vendors by counter-suing NEON Enterprise Software, IBM has now taken a swipe at the Oracle-Sun alliance by more the more conventional method of offering its own hardware and software to compete with their product.

IBM recently announced a new range of integrated systems for large-scale data analysis, which is clearly an attack on Oracle’s Exadata platform. Exadata first appeared in 2008, when Oracle partnered with Hewlett-Packard for the hardware. The 2009 V2 was meant to run on Sun servers. Exadata is an OLTP (OnLine Transaction Processing) database machine that Oracle claims is the world’s fastest machine for data warehousing and OLTP. By “world’s fastest” they, of course, mean faster than any IBM offering! As well as Oracle’s database and storage server software, it used Sun servers and FlashFire solid-state technology.

IBM’s response includes the pureScale application system plus Smart Analytics for System z and x86 machines. IBM is saying things like “deep compression” to reduce storage needs and an ability to handle “enormous amounts” of data. The pureScale application system comprises Power7 servers, WebSphere Application Server (WAS), and DB2 pureScale software. The Smart Analytics offering integrates Cognos BI and InfoSphere data warehousing software to handle business intelligence needs.

The mainframe Smart Analytics System 9600 is based on a System z10 Business Class (BC) mainframe with two logical partitions in a base configuration. There are six versions – the base 9600 has z/VM to manage the partitions and runs z/OS in one partition and Linux in another. The z/OS partition comes with DB2 and DB2 Utilities Suite for managing and tuning the database. The Linux part has Cognos 8 analytics and InfoSphere Warehouse.

Now in my opinion, this is a much better way of doing business – competition rather than litigation. Ordinary people – you and me, I mean – can then choose which they’d prefer, and go out and buy it (or rent it, or any of the other business models). With litigation, customers might be afraid that the products they have bought may become unlawful, or the company they bought from may be forced to cease trading, or they get so locked up in the court case that no further development of the product takes place. Now the cynical amongst you might suggest that that’s exactly what IBM hoped to get out of bringing in the lawyers. Maybe we really are returning to a climate of FUD (Fear, Uncertainty, and Doubt). Let’s hope not!