Welcome!

Linux Authors: Elizabeth White, ScriptRock Blog, Dale Kim, Carmen Gonzalez, Victoria Livschitz

Related Topics: Big Data Journal, Java, Linux, Open Source, SDN Journal

Big Data Journal: Article

Planet @Hadoop By @ABridgwater | @BigDataExpo [#BigData]

Hadoop has been called a foundational technology, rather than ‘just’ a database by some commentators

What Kinds of People Live on Planet Hadoop?

Labor market analytics firm Wanted Analytics recently assessed the market for technology professionals and found that demand for people with proficient levels of Hadoop expertise had skyrocketed by around 33% since last year - it is true, Hadoop is hard technology to master and the labor market is not exactly flooded with an over-abundance of skilled practitioners.

Hadoop has been called a foundational technology, rather than ‘just' a database by some commentators - this almost pushes it towards being an ‘environment' rather than it being a single software product... and this all goes towards making Hadoop even harder to master, many will agree.

Interestingly, we can essentially define Hadoop as a free Java-based ‘programming framework' that supports the processing of large data sets in a distributed computing environment with robust data processing and storage power.

Hadoop confusion
Is Hadoop a foundational technology, a data environment or programming framework? It is all of these things of course, but this may be where some of the challenge (and, specifically, the skills challenge) lies.

What kinds of people live on Planet Hadoop? Well, quite apart from being able to breath a noxious mixture of data, storage and Java-flavored oxygen, Hadoop native beings (we'll call them Hadoopers) are deeply innovative.

Hadoopers are the kind of people that are driven by an inherent desire to push the ‘analytics envelope' so far forward that it becomes an operational business process that in itself drives innovation - and therefore greater profits.

These are the sort of people who are prepared to help ‘industrialize the analytical process' and make it part of the way a firm operates from first principles. As Adrian Jones of SAS has put it, the analytics factory provides an approach that offers flexibility for analysts and a structured framework for governance.

"Your data preparation process should make it easy to quickly move between the two states. This step requires the creativity of the business combined with the process and operational efficiency of IT. Taking a factory approach to data preparation will remove this gray area from many organizations, reducing the conflict that comes through duplications and inefficiencies," writes Jones.

Hard-wired into Hadoop
As operational processes throughout the business start to get hard-wired into Hadoop, we see analytics deployed in a production state where it becomes part of the core workflow of all operations across the business - the truly analytical company starts to flourish from this point onward.

This Hadoop planet (or company, or environment, or biosphere) is populated by people with ability and aptitude of course; Hadoop is no place for dummies unfortunately... but what other traits to Hadoopers exhibit?

"Resourcefully creative with an ability to think on their feet," - would be a good way of describing these folk, i.e., they are capable of surveying the analytics landscape and knowing instinctively which datasets to combine, splice, analyze and focus on - but more than anything, they can perceive the kind of outcome that ‘might' be useful without actually knowing how it should initially be quantified and qualified.

A steely stoicism
Being able to convert the analytic process into an operational process requires an aptitude for practicality and a steely stoicism with the patience to wait and look for the right outcomes.

Really successful Hadoopers are corporate beasts with an appreciation for their company's regulatory processes and systems, i.e., there's not much point in producing great Hadoop analytics if it remains untamed analytics. Good corporate Hadoop citizens are capable of making Hadoop insights ‘universally accessible' so that every stakeholder in the organization gets the appropriate level of access.

As Fiona McNeill, Product Marketing, SAS, puts it, "The difference with analytically mature, innovative organizations, is that insights are universally accessible - whether you work in HR, finance, sales, logistics, marketing or services. The data is recognized as a corporate asset and analytical methods become intellectual property."

Nimble feet, fingers and foreheads
This discussion could go on and on. Good Hadoop pros exhibit many, many attributes, but one that is appropriate to finish with is nimbleness. Although this term ‘nimble' has become hackneyed, overused and overtired across the IT industry, we will bid to use it this one last time.

Hadoop is all about trial and error in terms of finding out what works best in relation to any particular analytics job. Nimble Hadoopers are unyielding in their ability to keep trying out analytics scenarios until they get what they need - or, perhaps more important, until they get what they want for the business.

These are some of the many elements currently populating the prosperous parts of Hadoop, whether we survive in this brave new world or not is dependent on our ability to be this kind of Hadooper - welcome to the new world.

This post is brought to you by SAS.

SAS is a leader in business analytics software and services and the largest independent vendor in the business intelligence market.

More Stories By Adrian Bridgwater

Adrian Bridgwater is a freelance journalist and corporate content creation specialist focusing on cross platform software application development as well as all related aspects software engineering, project management and technology as a whole.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
There's Big Data, then there's really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at Big Data Expo®, Hannah Smalltree, Director at Treasure Data, discussed how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other machines...
The Internet of Things (IoT) is making everything it touches smarter – smart devices, smart cars and smart cities. And lucky us, we’re just beginning to reap the benefits as we work toward a networked society. However, this technology-driven innovation is impacting more than just individuals. The IoT has an environmental impact as well, which brings us to the theme of this month’s #IoTuesday Twitter chat. The ability to remove inefficiencies through connected objects is driving change throughout every sector, including waste management. BigBelly Solar, located just outside of Boston, is trans...
Dale Kim is the Director of Industry Solutions at MapR. His background includes a variety of technical and management roles at information technology companies. While his experience includes work with relational databases, much of his career pertains to non-relational data in the areas of search, content management, and NoSQL, and includes senior roles in technical marketing, sales engineering, and support engineering. Dale holds an MBA from Santa Clara University, and a BA in Computer Science from the University of California, Berkeley.
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Advanced Persistent Threats (APTs) are increasing at an unprecedented rate. The threat landscape of today is drastically different than just a few years ago. Attacks are much more organized and sophisticated. They are harder to detect and even harder to anticipate. In the foreseeable future it's going to get a whole lot harder. Everything you know today will change. Keeping up with this changing landscape is already a daunting task. Your organization needs to use the latest tools, methods and expertise to guard against those threats. But will that be enough? In the foreseeable future attacks w...
As enterprises move to all-IP networks and cloud-based applications, communications service providers (CSPs) – facing increased competition from over-the-top providers delivering content via the Internet and independently of CSPs – must be able to offer seamless cloud-based communication and collaboration solutions that can scale for small, midsize, and large enterprises, as well as public sector organizations, in order to keep and grow market share. The latest version of Oracle Communications Unified Communications Suite gives CSPs the capability to do just that. In addition, its integration ...
Exploiting weak identity credentials is one of the most common elements of most attacks. Even attacks that seem unrelated to users, such as recent point of sale (POS) attacks at well-known retailers like the ones you have mentioned, can be traced to insecure passwords. Often, the same password is repeated for all organizations managed by the POS vendor, and once it is stolen, it essentially becomes a default password and the attackers can gain knowledge of the customer base.
The Internet of Things (IoT) promises to evolve the way the world does business; however, understanding how to apply it to your company can be a mystery. Most people struggle with understanding the potential business uses or tend to get caught up in the technology, resulting in solutions that fail to meet even minimum business goals. In his session at @ThingsExpo, Jesse Shiah, CEO / President / Co-Founder of AgilePoint Inc., showed what is needed to leverage the IoT to transform your business. He discussed opportunities and challenges ahead for the IoT from a market and technical point of vie...
Disruptive macro trends in technology are impacting and dramatically changing the "art of the possible" relative to supply chain management practices through the innovative use of IoT, cloud, machine learning and Big Data to enable connected ecosystems of engagement. Enterprise informatics can now move beyond point solutions that merely monitor the past and implement integrated enterprise fabrics that enable end-to-end supply chain visibility to improve customer service delivery and optimize supplier management. Learn about enterprise architecture strategies for designing connected systems tha...
As the Internet of Things unfolds, mobile and wearable devices are blurring the line between physical and digital, integrating ever more closely with our interests, our routines, our daily lives. Contextual computing and smart, sensor-equipped spaces bring the potential to walk through a world that recognizes us and responds accordingly. We become continuous transmitters and receivers of data. In his session at @ThingsExpo, Andrew Bolwell, Director of Innovation for HP's Printing and Personal Systems Group, discussed how key attributes of mobile technology – touch input, sensors, social, and ...
WebRTC defines no default signaling protocol, causing fragmentation between WebRTC silos. SIP and XMPP provide possibilities, but come with considerable complexity and are not designed for use in a web environment. In his session at @ThingsExpo, Matthew Hodgson, technical co-founder of the Matrix.org, discussed how Matrix is a new non-profit Open Source Project that defines both a new HTTP-based standard for VoIP & IM signaling and provides reference implementations.
From telemedicine to smart cars, digital homes and industrial monitoring, the explosive growth of IoT has created exciting new business opportunities for real time calls and messaging. In his session at @ThingsExpo, Ivelin Ivanov, CEO and Co-Founder of Telestax, shared some of the new revenue sources that IoT created for Restcomm – the open source telephony platform from Telestax. Ivelin Ivanov is a technology entrepreneur who founded Mobicents, an Open Source VoIP Platform, to help create, deploy, and manage applications integrating voice, video and data. He is the co-founder of TeleStax, a...
The worldwide cellular network will be the backbone of the future IoT, and the telecom industry is clamoring to get on board as more than just a data pipe. In his session at @ThingsExpo, Evan McGee, CTO of Ring Plus, Inc., discussed what service operators can offer that would benefit IoT entrepreneurs, inventors, and consumers. Evan McGee is the CTO of RingPlus, a leading innovative U.S. MVNO and wireless enabler. His focus is on combining web technologies with traditional telecom to create a new breed of unified communication that is easily accessible to the general consumer. With over a de...
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, demonstrated how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
Recurring revenue models are great for driving new business in every market sector, but they are complex and need to be effectively managed to maximize profits. How you handle the range of options for pricing, co-terming and proration will ultimately determine the fate of your bottom line. In his session at 15th Cloud Expo, Brendan O'Brien, Co-founder at Aria Systems, session examined: How time impacts recurring revenue How to effectively handle customer plan changes The range of pricing and packaging options to consider
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
Grow your business with enterprise wearable apps using SAP Platforms and Google Glass. SAP and Google just launched the SAP and Google Glass Challenge, an opportunity for you to innovate and develop the best Enterprise Wearable App using SAP Platforms and Google Glass and gain valuable market exposure. In his session at @ThingsExpo, Brian McPhail, Senior Director of Business Development, ISVs & Digital Commerce at SAP, outlined the timeline of the SAP Google Glass Challenge and the opportunity for developers, start-ups, and companies of all sizes to engage with SAP today.
The recent trends like cloud computing, social, mobile and Internet of Things are forcing enterprises to modernize in order to compete in the competitive globalized markets. However, enterprises are approaching newer technologies with a more silo-ed way, gaining only sub optimal benefits. The Modern Enterprise model is presented as a newer way to think of enterprise IT, which takes a more holistic approach to embracing modern technologies.