Getting started with Big Data

So, you’ve decided that you’re taking your organization down the route of Big Data, what components do you need? What are the available components that make Big Data work? Well. Let’s take a brief overview.

In terms of hardware, you’ll need lots of servers grouped into a very large cluster, with each server having its own internal disk drives. Ideally, you’d have Linux, but you might have Windows. And, of course, you could use Linux on System z if you have a mainframe.

You’re going to need a file system and that’s HDFS (Hadoop Distributed File System). Data in a Hadoop cluster gets broken down into smaller pieces that are called blocks, and these are distributed throughout the cluster. Any work on the data can then be performed on manageable pieces rather than on the whole mass of data.

Next you want a data store – and that’s HBase. HBase is an open source, non-relational, distributed database modelled after Google’s BigTable and is written in Java. It’s a column-oriented database management system (DBMS) that runs on top of HDFS. HBase applications are written in Java.

As a runtime, there’s MapReduce – a programming model for processing large data sets with a parallel, distributed algorithm on a cluster.

What about workload management, what options do you have for that? Your open source choices are ZooKeeper, Oozie, Jaql, Lucerne, HCatalog, Pig, and Hive. According to Apache, ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Similarly, according to Apache, Oozie is a workflow scheduler system to manage Hadoop jobs. Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions. Oozie Coordinator jobs are recurrent Oozie Workflow jobs triggered by time (frequency) and data availabilty. Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as MapReduce, Streaming MapReduce, Pig, Hive, Sqoop, and Distcp) as well as system specific jobs (such as Java programs and shell scripts). Jaql is primarily a query language for JavaScript Object Notation (JSON). It allows both structured and non-traditional data to be processed. Lucerne is an information retrieval software library from Apache that was originally created in Java. HCatalog is a table and storage management service for data created using Hadoop. Pig, also from Apache is a platform for analysing large data sets. It consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The structure of Pig programs allows substantial parallelization, which enables them to handle very large data sets. Finally on the list is Hive, which is a data warehouse system for Hadoop that facilitates easy data summarization, ad hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems.

So what are your integration options? Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. There’s also Sqoop, which is a tool designed for efficiently transferring bulk data between Hadoop and structured datastores such as relational databases.

And finally, is there an open source advanced analytic engine? There is and it’s called R. R is a programming language and a software suite used for data analysis, statistical computing, and data visualization. It is highly extensible and has object-oriented features and strong graphical capabilities. It is well-suited for modelling and running advanced analytics.

That will pretty much get you started and on your way. You may feel that you’d like more integration products, some form of administration, or some kind of visualization and discovery product. But this is where you need to go to specific vendors. I’m expecting to be at liberty to talk more about how IBM is looking at this in future blogs.

Fighting off the zombies!

Zombies are clearly very popular in books, on TV, and in movies and games. But they are spreading! Just recently, I’ve been hearing about zombie computers, zombie companies, and zombie everything else. So, I thought I’d take a look at this rise of the zombies!

So, let’s start with zombie computers. These look like ordinary Internet-connected computers, but they’re used to spread e-mail spam and launch distributed denial-of-service (DDOS) attacks. Unlike their movie counterparts, zombie computers don’t look any different. What’s happened is that the user has typically downloaded a virus or a trojan that has allowed a hacker to take control of their computer. The user continues, probably unaware, to use their machine, and the hacker can take control and get it to send spam e-mails, or try to access a designated Web site at a specified time. The only good news about a zombie laptop is that it can be revived (see a qualified technician to do so), and by using firewalls and antivirus software, further attacks can be prevented.

Quite different are zombie companies. These are companies that are struggling to stay afloat. They can just about afford the interest payments on their loans, but not much more. They are generating just about enough cash to service their debt, so the bank is not obliged to pull the plug on the loan. And so the company limps along, but it doesn’t have enough money to invest.

There are also zombie households. They have interest-only mortgages, which they can afford to pay the interest on, but they are unable to pay off the loan itself.

Then there’s zombie data. This is described as old forgotten data that you thought you’d deleted, but hadn’t. The trouble with this kind of data is that it could be accessed by hackers and could be used against you. People are likening it to data that you thought you’d thrown away, but someone sorts through your trash and finds it – and then uses it to perhaps access your system. We’re talking about old laptops that are given to charities without the hard drives being wiped, or data stored in the cloud in an account that isn’t much use any more – it’s forgotten, but not actually gone. Dormant files can be a danger!

Zombie programs are the programs that hackers use to gain access to your computer. They are often calls ‘bots’. And a series of linked zombie computers is a botnet.

I’m also sure that there are plenty of zombie programs sitting on mainframes and other platforms that were written years ago to perform important tasks and were never deleted. They’re sitting there – perhaps their existence is unknown to the current sys progs – waiting for someone to execute them. Perhaps, with all the changes that have taken place in the intervening years, they can do no harm. Or, perhaps they can cause mayhem! It might be worth checking that any of these zombie programs can’t come back and cause chaos.

You get zombie processes on Unix. These are processes that have completed execution, but they retain an entry in the process table – allowing the parent process to read its child’s exit status. Usually all entries are removed once the parent process has read the information it needs. You can identify a zombie using the ps command – it puts a ‘Z’ (for zombie) in the STAT column.

You can get zombie transactions in SqlTransaction code. With this, a zombie transaction is a transaction that cannot be committed (due to an unrecoverable error) but is still open.

COBOL is sometimes described as a zombie programming language because, no matter what else happens in programming languages, it’s always there – seemingly unkillable! In fact, IBM has recently announced the IBM Enterprise COBOL for z/OS V5.1 compiler.

Perhaps there are more zombies out there than you thought!
Next time, I’m definitely not talking about vampire and werewolf computing.

Big data – where are we?

At first, people would enter information into their computers, then print it off if they wanted to share the data. Then we had networks and people could electronically share data – and then others could add to it. Pretty much all the data – even in the largest IMS database – had been entered by people or calculated from data entered by people.

But more recently, things have changed. Information stored on computers has come from other sources, for example card readers, CCTV cameras, traffic flow sensors, etc, etc. Almost any device can be given an IP address, connected to a network, and used as a source of data. All these ‘things’, that can and are being connected, has led to the use of the phrase: ‘the Internet of things’. Perhaps not the most precise description, but it indicates that the Internet is being used as a way of getting information from devices – rather than waiting for a human to type in the data.

The other development that we’re all familiar with is the growth in cloud computing. What that means is devices are connected to a nebulous source of storage and processing power. Mainframers, who have been around the block a few times, feel quite happy with this model of dumb terminals connected to some giant processing device that is some distance away and not necessarily visible to the users of the dumb terminals. This is what mainframe computing was like (and still is for some users!). Other computer professionals will recognize this as another version of the client/server model that was once so fashionable.

By having so many sources of data input, you have security and storage issues, but, perhaps more importantly, you have issues about what to do with the data. It’s almost like a person with OCD hoarding old newspaper that they never look at but can’t throw away. What can you do with these vast amounts of data?

The answer is Hadoop. According to the Web site at http://hadoop.apache.org/: “The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.”

So which companies are experienced with Hadoop? Cloudera was probably the best known in the field up until recently. Other companies you may not have heard of are MapR and Hortonworks. Companies you will be familiar with are EMC and VMware who have spun off a company called Pivotal. And there’s Intel, and there’s IBM.

Let’s have a quick look at what’s out there. Apache Hive was developed by Facebook, but is now Open Source. Dremel (from Google) is published, but not yet available. Apache Drill is based on Dremel, but is still in the incubation stage. Cloudera’s Impala was inspired by Dremel. IBM’s offering is Big SQL. Hive is a data warehouse infrastructure built on top of Hadoop. It converts queries into MapReduce jobs. Impala’s SQL query system for Hadoop is Open Source. It uses C++ rather than Java. It doesn’t use MapReduce. Impala only works with Cloudera’s Distribution of Hadoop (CDH).

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml, and Delphi and other languages.

IBM’s Big SQL is a currently a technology preview. It supports SQL, and JDBC and ODBC client drivers. IBM’s distribution of Hadoop is called BigInsights. Big SQL is similar to Hive and they can cross query. Point query is used for small queries rather than MapReduce. It supports more datatypes than Hive.

So, you can see that there’s lot’s to learn about Hadoop, and I’m sure we’ll be hearing a lot more about BigInsights and Big SQL. My advice is, if you’re looking for a career path, companies are going to need experienced Hadoop people – so get some!

OmniPage Ultimate 19 - review

There was a time when I used to scan in a lot of documents. You’d go to an event or conference and people would give you handouts. If they were any use, you might scan in the text so you had it easily available. These days, there’s less and less need for that because people don’t print off 50 copies of a document, they give you a link to an online PDF. So if you do have scanning software, you want it to be pretty amazing.

Of course, HP sells lots of ‘all-in-one’ scanner/copier/printer units to the home market. So those people are going to want OCR (Optical Character Recognition) software to convert scanned documents to something that is useful. I’m sure many parents can help their children with homework by scanning in text and images and reproducing that in a different format for school’s consumption.

So, let’s suppose you go out and purchase the latest version of Nuance’s OmniPage Ultimate, what do you get? Well, to start with you get an installation that irritatingly launches Google Chrome in an attempt to register the software. By the time I’d cancelled that and put the address into Firefox, my software had given up registering and said it would try again in 7 days. Not an auspicious start.

I then launched the software expecting some user-friendly front end that would talk me through what I wanted to do. ’Fraid not. I connected the software to my scanner easily enough. I clicked around a lot expecting to find some way to scan in a document. After 10 minutes I just gave up – it didn’t seem to want to do that. In fact, it isn’t designed to do that. So I scanned in a document in my usual way and saved it as a JPG. I then opened that in OmniPage. It was a mixture of text and images. And the OCR was perfect. OK it wasn’t the hardest document to read, but it was still perfect – down to the last semi-colon. I then tried the speech output – trying out the different voices available – and that was well done. It read the text nicely. I could listen on any mobile device (tablet or smartphone) to it reading – so long as the device supported mp3 files. I later converted a long PDF document, I’d been sent, to mp3 and listened to it on my tablet – brilliant.

A program that comes with OmniPage Ultimate is Launchpad. This looks a bit Windows 8ish and uses tiled action buttons to speed up the process of converting documents and sending them to any destination. This is obviously an attempt to freshen up the user interface – and it works.

What makes Ultimate more than just an OCR engine is that it can convert PDFs and digital photos of text into files that users can edit/search/share. Plus, it can convert those into mp3 speech audio files that can stored on mobile devices (as I mentioned above). It means that you can listen to a document on the go, rather than trying to read it. File types it can save as includes: Microsoft Word, Excel, and PowerPoint 2007, PDF, HTML, and Corel WordPerfect.

One clever new feature is that users can convert documents so they can be read with eBook applications on the Android tablets, Microsoft Surface, iPad,Nook Simple Touch, Nook Tablet, Sony Reader PRS-350/PRS-650/PRS-950, Kobo eReader, Kobo Touch and Kobo Arc.

The press release also point out that Ultimate can convert “documents stored in Windows Live SkyDrive, GoogleDocs, Evernote, Box, Dropbox, and many more when integrated with Nuance Cloud Connector. OmniPage connects with Microsoft SharePoint and DMS, and features new support for Open Text Hummingbird Connector, and Autonomy-Interwoven iManage Connector”.

Also of interest to some people will be the fact that Ultimate is accessible in “more than 100 languages. OmniPage Ultimate includes the recognition of languages based on the Latin, Greek, and Cyrillic alphabets as well as Chinese, Japanese and Korean, languages enabling users from around the world to access, edit and manage documents”.

So, apart from the registration annoyance and the old-fashioned user interface (although Launchpad helps modernize things), it is a very powerful piece of software with very good character recognition success (I tested a fair few scans and PDFs before I wrote this) that makes documents available for editing and sharing. The OmniPage file can be exported as mp3 as well as HTML and the usual Office programs. I must admit, the product grew on me, the more I used it and the more familiar I became with it.

You can find out more at www.nuance.com/for-business/by-product/omnipage/ultimate/index.htm.

Carrots, sticks, and managing mainframe staff

So, how do you ensure that your staff work to the best of their ability? Because, no matter how fast your mainframe is, if your expert staff aren’t motivated and engaged, then things will move forward slowly. Upgrades, fixes, ZAPs, etc will be reluctantly and perhaps inappropriately applied. And performance generally will be poor.

Research has generally shown that ‘carrots’ are better motivators in terms of encouraging good behaviour from staff, and ‘sticks’ are not so good at motivating staff. So praising good work by staff seems to be a better way of getting them to do what you want than criticism of their work or blaming them for failures. But simply saying “well done” may have its pitfalls!

Management mostly use the operant conditioning model of learning. In this, reinforcement encourages a particular behaviour and punishment discourages the behaviour. Simple! From that you’d assume that incentives or rewards and praise in the workplace would be successful in getting the best results from staff. However, research indicates that rewards are effective at producing only temporary compliance – they don’t seem to produce lasting change in attitudes or behaviour. And research shows that people who expect to receive a reward don’t perform as well as those who expect nothing! So people expecting praise for doing their job, will, in the fullness of time, do it less well!

One technique that many classroom teachers use to ensure their children behave correctly is assertive discipline. With this technique, clear rules are laid out, and then children are rewarded (praised) for following the rules. For example, one group of children are sitting quietly waiting for the teacher while another group are chatting. Rather than telling the noisy group to settle down, the teacher will say something like: “Look how well these children are sitting”. The inappropriately behaving group learn that to get attention and approval they need to follow the rules.

From that, we might learn that in the work place, bosses should recognize and comment on desirable behaviour from their staff. Basically, they should be offering praise for people doing the right thing.

Attachment parenting takes a different view of praise – and, although it’s obviously aimed at young children, it has an application with staff. So, rather than saying “good boy”, you say what you can see and feel. So, you might say, “you sorted out your pencils and crayons and put them in separate boxes. That’s impressive organization”. The idea behind it is that the praise of “good boy” could easily be taken away by saying “naughty boy” at a different time. Descriptive praise can’t be taken away. So, practice saying things like: “I see you made the IMS system perform more efficiently. That takes determination.”

But what makes a person behave in a particular way? What makes them stay late until piece of work is completed, or go above and beyond their usual role? It seems there are intrinsic motivators and extrinsic motivators. Extrinsic motivation comes because you expect a reward when the task is complete. Intrinsic motivation comes because the task itself is motivating. Intrinsic motivation is really all about autonomy. You control what you do and when you do it. But, once you start rewarding people who are intrinsically motivated, it demotivates them! There is, apparently, a way round this, you can randomly reward them. So again, praising people seems to have a surprising and opposite effect to what you’d expect.

Mark Tyrrell in his blog “Why telling people how wonderful they are isn’t always a good idea” at http://www.unk.com/blog/dangers-of-excessive-praise/?utm_source=clearthinking&utm_medium=email&utm_content=171&utm_campaign=Clear%2BThinking%2Bnewsletter looks further at why saying, “well done” isn’t such a good idea.

He reminds us of a study, published in 1998 in the Journal of Personality and Social Psychology, in which children were rewarded for simply ‘doing their own thing’ – drawing, playing, and so on. But when the rewards were discontinued, the children tended to lose interest in their preferred activity.

His conclusions were:

• Don’t over-praise people (or yourself) for doing stuff they should be doing anyway. Sure, it’s good I don’t go around stealing from people, but it should also be my normal behaviour. So be discerning and selective with praise. Ask yourself: “Am I being praised (or praising someone else) for normal human behaviour?”
• Focus on the normality of the desirable behaviour rather than implying that it’s exceptional and therefore not something that can happen often.
• Don’t expect praise for everything and don’t always praise others, or you’ll be fostering praise dependency and an inability to pursue excellence for its own sake. If you don’t praise all the time, the praise you do offer has more potency.
• Focus on what is actually within a person’s control (like hard work) rather on nebulous and essentially meaningless characteristics. “You have worked really hard!” implies that success was something they could consciously influence. Telling someone they are “fabulous” for completing something gives them no meaningful information about their input.

He goes on to say: “Sugar is nice, but a little goes a long way and too much can really screw you up.

“The right kind of praise at the right time and in the right quantity can help you develop the habit of excellence, but a diet of uncontrolled praise won’t do you or anyone else any favours.”

It’s interesting to think how this can be applied to staff working on the mainframe, network, and distributed platforms to get the best out of them.

Welcome to the red team!

You may not know that ‘red teaming’ refers to the practice of “viewing a problem from an adversary or competitor’s perspective. The goal of most red teams is to enhance decision making, either by specifying the adversary’s preferences and strategies or by simply acting as a devil’s advocate. Red teaming may be more or less structured, and a wide range of approaches exists. In the past several years, red teaming has been applied increasingly to issues of security, although the practice is potentially much broader. Business strategists, for example, can benefit from weighing possible courses of action from a competitor’s point of view.” That definition comes from the Red Team Journal at http://redteamjournal.com/about/red-teaming-and-alternative-analysis/.

One thing that red teams are often asked to do these days is test IT security. The red team will try to infiltrate a company’s IT system in order to identify any previously unknown vulnerabilities. It seems that one of the best ways to get into a system is to be the first to find a new vulnerability in the software that no-one else has spotted. This ‘zero day’ vulnerability can be used to get malware of some kind into an organization, and, from then on, the red team own the IT system. And that’s why it’s a good idea to pay a team of experts rather than wake up one day and find the bad guys have found their way into your IT infrastructure.

Basically, that small piece of malware can be used by the red team to gain access to the network. And from there they can gain access to any documents or databases and download whatever information they want. If you’re company is a bank, they could find a way to steal money. And a lot of the time, no-one would know it was happening until it’s too late.

Internet Explorer has been in the press over the years for the number of vulnerabilities that it once had, but nowadays, Java is a prime target for red teams because Java is meant to run on 3 million devices – providing what’s called a large ‘attack surface’. Stack walking refers to the way that the different components of an IT platform exchange information about security privileges. This makes it an ideal target.

Attacking the software is getting harder these days, but there’s one component of an organization’s computer system that is always potentially vulnerable – and that’s the people who use the computers. In the past there were stories of dumpster diving – where people would look through dumpsters and rubbish bins for information on paper that was thrown away. Nowadays, most companies collect and destroy paper, so that can’t happen. Even so, walking around a building a visitor can still find passwords on Post-It notes stuck to screens. There’s also a technique called spearphishing that can be used. In this, a seemingly legitimate e-mail contains a malicious link or attachment. Once a person clicks on the link or opens the attachment, the malware is on the system. Another technique is to send infected memory sticks to staff, who often plug them in to see what’s on them, and, again, the malware strikes!

Red team members can now use social media to find the names of staff as well as details of their experience, so that e-mails and phone calls from the red team can sound quite legitimate. Part of the answer is SIEM (Security Information and Event Management) solutions. These provide real-time analysis of security alerts generated by network hardware and applications. SIEM solutions come as software, appliances, or managed services, and are also used to log security data and generate reports for compliance purposes.

The other part of the solution is education of staff so that they don’t insert memory sticks or click on attachments from unknown sources. But often, the best way to get access to corporate data is to find a disgruntled employee. So maybe another part of the solution is to ensure that staff are happy – that terms and conditions are going to avoid people feeling disgruntled. And if they are, then policies and procedures must be in place to manage that situation. And that’s not so easy with a large organization.

Mainframes are mostly used by large organizations – which obviously puts them at risk from unhappy employees. The risk is increased because most mainframe sites also use other platforms – PCs etc. And there is a new and huge security risk with BYOD. The red team could, perhaps, get a piece of malware onto someone’s tablet, which then gets connected to network, which then starts opening security doors all the way to the mainframe.

You may feel your data isn’t important enough to warrant the employment of a red team to test out any exposure to vulnerabilities you might have. But most organizations can learn from the types of vulnerability red teams exploit, and take steps to ensure that they are not at risk from them.

Using social media - Badminton

Badminton horse trials are taking place over the bank holiday weekend from Thursday 2 May to Monday 6 May this year. Badminton is one of those big events that horse lovers from all over the world like to attend – and so do I. And this week, I thought it would be interesting to have a look at how it does its IT. Bear in mind that it has lots of scores coming from different sources, as well as lots of photographs and film, that all need to be collated and shared with the fans who are onsite and off.

Badminton takes place in the park of Badminton House, in Gloucestershire GL9 1DF, England. The house and grounds are owned by the Duke of Beaufort. Badminton has been going since 1949 and is one of the top three four-star rated events (along with Burghley Horse Trials and Rolex Kentucky Three Day). It’s also one of only six annual Concours Complet International (CCI) Four Star events. Three-day eventing involves dressage, cross country, and show jumping.

I spoke to Dominic Sancto who gave me an overview of how things are set up at Badminton – as far IT is concerned.

This year, scores are entered into PCs and these results are wifi’ed across to Windows servers, where a bespoke database collects them all. From there, the results are written out to flat files and distributed accordingly. That means that the flat files are used to update the Web site and result screens, as well as the Badminton app.

There are three versions of the Badminton app, one for Android devices, one for iPhones, and one for iPads. The app gives details of the timetable and start times, as well as results. It also gives news and social information, video and audio, a photo gallery, the course, the riders, visitor information, and ‘grassroots’ – the BE 90 Class riders (that’s a lower group than the rest of the competitors, but everyone’s got to start somewhere!).

As well as the phone app and the Web site, Badminton’s social media has a Facebook page, Twitter, an RSS feed, Picasa, and YouTube. The Web site at www.badminton-horse.co.uk/ 
 has all the information that appears on the app and more. There’s information, tickets/hospitality, horses & riders, news, shopping village, gallery, and an archive. There is just so much information – it’s excellent.

When it comes to Twitter, there are seven members of the Badminton crew tweeting information, news , and comments about what’s going. You can find them at twitter.com/bht_office/badminton-crew. On Facebook, they’re at www.facebook.com/BadmintonHorseTrials. There were plenty of photos and comments before the event even started, and nearly 47,000 likes by last Monday. Not surprisingly, there are even more photos (including from previous years) on Picasa at picasaweb.google.com/baddershorsetrials. Like most organizations, Badminton IT has moved its movies to YouTube to avoid their own servers being overloaded. You can see plenty of videos at www.youtube.com/baddershorsetrials.

There’s also Radio Badminton on 106.1 FM – and you can buy ear-size radios so you can listen to everything that’s going on in the arena as you walk round the massive shopping village or follow the cross country course.

The whole of the IT is very well suited to the needs of the thousands of people who will be visiting Badminton this weekend. See you there!