Click here to close now.





















Welcome!

Apache Authors: Pat Romanski, Jim Scott, Jnan Dash, Craig Lowell, Liz McMillan

Related Topics: @DevOpsSummit, IBM Cloud, Apache

@DevOpsSummit: Article

IBM’s Big Commitment to Apache Spark | @CloudExpo #DevOps #Microservices

It will offer Apache Spark as a service on Bluemix

Last June IBM made a serious commitment to the future of Apache Spark with a series of initiatives:

  • It will offer Apache Spark as a service on Bluemix (Bluemix is an implementation of IBM's Open Cloud Architecture based on Cloud Foundry, an open source Platform as a Service (PaaS). Bluemix delivers enterprise-level services that can easily integrate with your cloud applications without you needing to know how to install or configure them.
  • It committed to include 3500 researchers to work on Spark-related projects.
  • It will donate IBM SystemML (its machine learning language and libraries) to Apache Spark open source

The question is why this move by IBM?

First let us look at what is Apache Spark? Developed at UC Berkeley's AMPLab, Spark gives us a comprehensive, unified framework to manage big data processing requirements with a variety of data sets that are diverse in nature (text data, graph data etc) as well as the source of data (batch v. real-time streaming data). Spark enables applications in Hadoop clusters to run up to 100 times faster in memory and 10 times faster even when running on disk. In addition to Map and Reduce operations, it supports SQL queries, streaming data, machine learning and graph data processing. Developers can use these capabilities stand-alone or combine them to run in a single data pipeline use case. In other words, Spark is the next-generation of Hadoop (came with its batch pedigree and high latency).

With other solutions for real-time analytics via in-memory processing such as RethinkDB, an ambitious Redis project or commercial in-memory SAP Hana, IBM needed a competitive offering. Other vendors betting on Spark range from Amazon to Zoomdata. IBM will run its own analytics software on top of Spark, including SystemML for machine learning, SPSS, and IBM Streams.

At this week's Strata conference, several companies like Uber described how they have deployed Spark all the way for speedy real-time analytics. 

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

@ThingsExpo Stories
For manufacturers, the Internet of Things (IoT) represents a jumping-off point for innovation, jobs, and revenue creation. But to adequately seize the opportunity, manufacturers must design devices that are interconnected, can continually sense their environment and process huge amounts of data. As a first step, manufacturers must embrace a new product development ecosystem in order to support these products.
SYS-CON Events announced today that Ericsson has been named “Silver Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Ericsson strives to connect everyone, wherever they may be. Because by being connected, people can take part in the emerging global collaboration that is the Networked Society – a society in which every person and every industry is empowered to reach their full potential.
The IoT's basic concept of collecting data from as many sources possible to drive better decision making, create process innovation and realize additional revenue has been in use at large enterprises with deep pockets for decades. So what has changed? In his session at @ThingsExpo, Prasanna Sivaramakrishnan, Solutions Architect at Red Hat, will discuss the impact commodity hardware, ubiquitous connectivity, and innovations in open source software are having on the connected universe of people, things and information in the IoT.
The broad selection of hardware, the rapid evolution of operating systems and the time-to-market for mobile apps has been so rapid that new challenges for developers and engineers arise every day. Security, testing, hosting, and other metrics have to be considered through the process. In his session at Big Data Expo, Walter Maguire, Chief Field Technologist, HP Big Data Group, at Hewlett-Packard, will discuss the challenges faced by developers and a composite Big Data applications builder, focusing on how to help solve the problems that developers are continuously battling.
With the exponential growth of network traffic slowing down data transmission, companies are looking for solutions. Recently, a solution has emerged that can help improve your data speed with data centers on the edge. These micro data center solutions can simplify the lives of many data center owners and operators because they are self-contained, secure computing environments, assembled in a factory and shipped in one enclosure which includes all the necessary power, cooling, security, and management tools. Their flexibility opens up a wave of new applications, made possible through reduced la...
With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now all corporate assets – people, objects, and spaces – can share information about themselves and thei...
The IoT's basic concept of collecting data from as many sources possible to drive better decision making, create process innovation and realize additional revenue has been in use at large enterprises with deep pockets for decades. So what has changed? In his session at @ThingsExpo, Prasanna Sivaramakrishnan, Solutions Architect at Red Hat, will discuss the impact commodity hardware, ubiquitous connectivity, and innovations in open source software are having on the connected universe of people, things and information in the IoT.
Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, and June 7-9, 2016 at Javits Center, New York City, is co-located with Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
Most of the IoT Gateway scenarios involve collecting data from machines/processing and pushing data upstream to cloud for further analytics. The gateway hardware varies from Raspberry Pi to Industrial PCs. The document states the process of allowing deploying polyglot data pipelining software with the clear notion of supporting immutability. In his session at @ThingsExpo, Shashank Jain, a development architect for SAP Labs, will discuss the objective, which is to automate the IoT deployment process from development to production scenarios using Docker containers.
Container technology is shaping the future of DevOps and it’s also changing the way organizations think about application development. With the rise of mobile applications in the enterprise, businesses are abandoning year-long development cycles and embracing technologies that enable rapid development and continuous deployment of apps. In his session at DevOps Summit, Kurt Collins, Developer Evangelist at Built.io, examines how Docker has evolved into a highly effective tool for application delivery by allowing increasingly popular Mobile Backend-as-a-Service (mBaaS) platforms to quickly crea...
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists will address the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true change and transformation possible.
SYS-CON Events announced today that Colovore, the Bay Area’s leading provider of scalable, high-density colocation solutions, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. With power densities of 20 kW per rack and a pay-by-the-kW pricing model, Colovore is Silicon Valley’s premier provider of flexible, high-density colocation solutions. Our 9 MW facility has the power and cooling your servers need, and our team has decades of experience managing web infrastructure. We are optimized to ...
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will discuss what things are the most important, which will have the most profound effect on the world, and what should we expect to see over the next couple of years.
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
Beacon technology offers a new way for companies to deliver value to their customers by enabling micro-location based interaction at every physical touch point. This knowledge opens up a new playground for creating personalized services and inventing new experiences between a company and its customers, but how exactly can organizations monetize this? For the past two years we have heard about mobile notifications and messaging, but this has not taken off and could be considered spamming.
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi’s VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context wi...
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of (at least) three separate application components: the software embedded in the device, the back-end service, and the mobile application for the end user’s controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target – this makes the integration of these separate pipelines and the coordination of software updates for...
In his session at @ThingsExpo, Ben Bromhead, CTO of Instaclustr, will walk you through the basics of building an IoT-based platform leveraging Cassandra, Spark and Kafka. This session is aimed at developers, admins and DevOps engineers who have to build, run and maintain high performance IoT platforms as well as data scientists/engineers who are sick of ETL and want to work with the most up to date information.
From Fitbits, to connected cars, to sensors that water crops, making decisions is no longer enough, now you need to make decisions in context. To bring your next great IoT decision to life you need to be able to extend and connect beyond your traditional enterprise. But with thousands if not millions of different devices all with their own technologies, standards and security how do you connect with the right ones effectively. And technology is only one aspect of the challenge; how do you create the right business model to drive value from your solution? An IoT solution that doesn’t connect to...