For manufacturers, the Internet of Things (IoT) represents a jumping-off point for innovation, jobs, and revenue creation. But to adequately seize the opportunity, manufacturers must design devices that are interconnected, can continually sense their environment and process huge amounts of data.
As a first step, manufacturers must embrace a new product development ecosystem in order to support these products. | By Jnan Dash | Article Rating: |
|
| October 24, 2015 04:00 AM EDT | Reads: |
418 |

Last June IBM made a serious commitment to the future of Apache Spark with a series of initiatives:
- It will offer Apache Spark as a service on Bluemix (Bluemix is an implementation of IBM's Open Cloud Architecture based on Cloud Foundry, an open source Platform as a Service (PaaS). Bluemix delivers enterprise-level services that can easily integrate with your cloud applications without you needing to know how to install or configure them.
- It committed to include 3500 researchers to work on Spark-related projects.
- It will donate IBM SystemML (its machine learning language and libraries) to Apache Spark open source
The question is why this move by IBM?
First let us look at what is Apache Spark? Developed at UC Berkeley's AMPLab, Spark gives us a comprehensive, unified framework to manage big data processing requirements with a variety of data sets that are diverse in nature (text data, graph data etc) as well as the source of data (batch v. real-time streaming data). Spark enables applications in Hadoop clusters to run up to 100 times faster in memory and 10 times faster even when running on disk. In addition to Map and Reduce operations, it supports SQL queries, streaming data, machine learning and graph data processing. Developers can use these capabilities stand-alone or combine them to run in a single data pipeline use case. In other words, Spark is the next-generation of Hadoop (came with its batch pedigree and high latency).
With other solutions for real-time analytics via in-memory processing such as RethinkDB, an ambitious Redis project or commercial in-memory SAP Hana, IBM needed a competitive offering. Other vendors betting on Spark range from Amazon to Zoomdata. IBM will run its own analytics software on top of Spark, including SystemML for machine learning, SPSS, and IBM Streams.
At this week's Strata conference, several companies like Uber described how they have deployed Spark all the way for speedy real-time analytics. ![]()
Published October 24, 2015 Reads 418
Copyright © 2015 SYS-CON Media, Inc. — All Rights Reserved.
Syndicated stories and blog feeds, all rights reserved by the author.
More Stories By Jnan Dash
Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.
For manufacturers, the Internet of Things (IoT) represents a jumping-off point for innovation, jobs, and revenue creation. But to adequately seize the opportunity, manufacturers must design devices that are interconnected, can continually sense their environment and process huge amounts of data.
As a first step, manufacturers must embrace a new product development ecosystem in order to support these products. Oct. 25, 2015 03:00 PM EDT Reads: 221 |
By Elizabeth White SYS-CON Events announced today that Ericsson has been named “Silver Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
Ericsson strives to connect everyone, wherever they may be. Because by being connected, people can take part in the emerging global collaboration that is the Networked Society – a society in which every person and every industry is empowered to reach their full potential.Oct. 25, 2015 02:00 PM EDT Reads: 451 |
By Carmen Gonzalez The IoT's basic concept of collecting data from as many sources possible to drive better decision making, create process innovation and realize additional revenue has been in use at large enterprises with deep pockets for decades. So what has changed?
In his session at @ThingsExpo, Prasanna Sivaramakrishnan, Solutions Architect at Red Hat, will discuss the impact commodity hardware, ubiquitous connectivity, and innovations in open source software are having on the connected universe of people, things and information in the IoT.Oct. 25, 2015 02:00 PM EDT Reads: 224 |
By Yeshim Deniz The broad selection of hardware, the rapid evolution of operating systems and the time-to-market for mobile apps has been so rapid that new challenges for developers and engineers arise every day. Security, testing, hosting, and other metrics have to be considered through the process.
In his session at Big Data Expo, Walter Maguire, Chief Field Technologist, HP Big Data Group, at Hewlett-Packard, will discuss the challenges faced by developers and a composite Big Data applications builder, focusing on how to help solve the problems that developers are continuously battling.Oct. 25, 2015 02:00 PM EDT Reads: 239 |
By Elizabeth White With the exponential growth of network traffic slowing down data transmission, companies are looking for solutions. Recently, a solution has emerged that can help improve your data speed with data centers on the edge. These micro data center solutions can simplify the lives of many data center owners and operators because they are self-contained, secure computing environments, assembled in a factory and shipped in one enclosure which includes all the necessary power, cooling, security, and management tools. Their flexibility opens up a wave of new applications, made possible through reduced la...Oct. 25, 2015 02:00 PM EDT Reads: 321 |
By Elizabeth White With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now all corporate assets – people, objects, and spaces – can share information about themselves and thei...Oct. 25, 2015 02:00 PM EDT Reads: 353 |
By Elizabeth White Oct. 25, 2015 02:00 PM EDT Reads: 585 |
By Carmen Gonzalez Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, and June 7-9, 2016 at Javits Center, New York City, is co-located with Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.Oct. 25, 2015 01:30 PM EDT Reads: 301 |
By Liz McMillan Most of the IoT Gateway scenarios involve collecting data from machines/processing and pushing data upstream to cloud for further analytics. The gateway hardware varies from Raspberry Pi to Industrial PCs. The document states the process of allowing deploying polyglot data pipelining software with the clear notion of supporting immutability.
In his session at @ThingsExpo, Shashank Jain, a development architect for SAP Labs, will discuss the objective, which is to automate the IoT deployment process from development to production scenarios using Docker containers.Oct. 25, 2015 01:00 PM EDT Reads: 276 |
By Pat Romanski Container technology is shaping the future of DevOps and it’s also changing the way organizations think about application development. With the rise of mobile applications in the enterprise, businesses are abandoning year-long development cycles and embracing technologies that enable rapid development and continuous deployment of apps.
In his session at DevOps Summit, Kurt Collins, Developer Evangelist at Built.io, examines how Docker has evolved into a highly effective tool for application delivery by allowing increasingly popular Mobile Backend-as-a-Service (mBaaS) platforms to quickly crea...Oct. 25, 2015 01:00 PM EDT Reads: 239 |
By Pat Romanski Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible.
In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists will address the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true change and transformation possible.Oct. 25, 2015 01:00 PM EDT Reads: 386 |
By Elizabeth White SYS-CON Events announced today that Colovore, the Bay Area’s leading provider of scalable, high-density colocation solutions, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
With power densities of 20 kW per rack and a pay-by-the-kW pricing model, Colovore is Silicon Valley’s premier provider of flexible, high-density colocation solutions. Our 9 MW facility has the power and cooling your servers need, and our team has decades of experience managing web infrastructure. We are optimized to ...Oct. 25, 2015 01:00 PM EDT Reads: 291 |
By Liz McMillan The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today. Oct. 25, 2015 12:30 PM EDT Reads: 324 |
By Liz McMillan The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas.
In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will discuss what things are the most important, which will have the most profound effect on the world, and what should we expect to see over the next couple of years.Oct. 25, 2015 12:30 PM EDT Reads: 288 |
By Pat Romanski Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.Oct. 25, 2015 12:00 PM EDT Reads: 275 |
By Pat Romanski Oct. 25, 2015 12:00 PM EDT Reads: 169 |
By Yeshim Deniz Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability.
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi’s VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context wi...Oct. 25, 2015 11:00 AM EDT Reads: 223 |
By Liz McMillan Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of (at least) three separate application components: the software embedded in the device, the back-end service, and the mobile application for the end user’s controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target – this makes the integration of these separate pipelines and the coordination of software updates for...Oct. 25, 2015 11:00 AM EDT Reads: 364 |
By Pat Romanski In his session at @ThingsExpo, Ben Bromhead, CTO of Instaclustr, will walk you through the basics of building an IoT-based platform leveraging Cassandra, Spark and Kafka. This session is aimed at developers, admins and DevOps engineers who have to build, run and maintain high performance IoT platforms as well as data scientists/engineers who are sick of ETL and want to work with the most up to date information.Oct. 25, 2015 10:00 AM EDT Reads: 426 |
By Elizabeth White From Fitbits, to connected cars, to sensors that water crops, making decisions is no longer enough, now you need to make decisions in context. To bring your next great IoT decision to life you need to be able to extend and connect beyond your traditional enterprise. But with thousands if not millions of different devices all with their own technologies, standards and security how do you connect with the right ones effectively. And technology is only one aspect of the challenge; how do you create the right business model to drive value from your solution? An IoT solution that doesn’t connect to...Oct. 25, 2015 09:00 AM EDT Reads: 253 |

SYS-CON Events announced today that Ericsson has been named “Silver Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
Ericsson strives to connect everyone, wherever they may be. Because by being connected, people can take part in the emerging global collaboration that is the Networked Society – a society in which every person and every industry is empowered to reach their full potential.
The IoT's basic concept of collecting data from as many sources possible to drive better decision making, create process innovation and realize additional revenue has been in use at large enterprises with deep pockets for decades. So what has changed?
In his session at @ThingsExpo, Prasanna Sivaramakrishnan, Solutions Architect at Red Hat, will discuss the impact commodity hardware, ubiquitous connectivity, and innovations in open source software are having on the connected universe of people, things and information in the IoT.
The broad selection of hardware, the rapid evolution of operating systems and the time-to-market for mobile apps has been so rapid that new challenges for developers and engineers arise every day. Security, testing, hosting, and other metrics have to be considered through the process.
In his session at Big Data Expo, Walter Maguire, Chief Field Technologist, HP Big Data Group, at Hewlett-Packard, will discuss the challenges faced by developers and a composite Big Data applications builder, focusing on how to help solve the problems that developers are continuously battling.
With the exponential growth of network traffic slowing down data transmission, companies are looking for solutions. Recently, a solution has emerged that can help improve your data speed with data centers on the edge. These micro data center solutions can simplify the lives of many data center owners and operators because they are self-contained, secure computing environments, assembled in a factory and shipped in one enclosure which includes all the necessary power, cooling, security, and management tools. Their flexibility opens up a wave of new applications, made possible through reduced la...
With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now all corporate assets – people, objects, and spaces – can share information about themselves and thei...
Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, and June 7-9, 2016 at Javits Center, New York City, is co-located with Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
Most of the IoT Gateway scenarios involve collecting data from machines/processing and pushing data upstream to cloud for further analytics. The gateway hardware varies from Raspberry Pi to Industrial PCs. The document states the process of allowing deploying polyglot data pipelining software with the clear notion of supporting immutability.
In his session at @ThingsExpo, Shashank Jain, a development architect for SAP Labs, will discuss the objective, which is to automate the IoT deployment process from development to production scenarios using Docker containers.
Container technology is shaping the future of DevOps and it’s also changing the way organizations think about application development. With the rise of mobile applications in the enterprise, businesses are abandoning year-long development cycles and embracing technologies that enable rapid development and continuous deployment of apps.
In his session at DevOps Summit, Kurt Collins, Developer Evangelist at Built.io, examines how Docker has evolved into a highly effective tool for application delivery by allowing increasingly popular Mobile Backend-as-a-Service (mBaaS) platforms to quickly crea...
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible.
In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists will address the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true change and transformation possible.
SYS-CON Events announced today that Colovore, the Bay Area’s leading provider of scalable, high-density colocation solutions, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
With power densities of 20 kW per rack and a pay-by-the-kW pricing model, Colovore is Silicon Valley’s premier provider of flexible, high-density colocation solutions. Our 9 MW facility has the power and cooling your servers need, and our team has decades of experience managing web infrastructure. We are optimized to ...
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas.
In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will discuss what things are the most important, which will have the most profound effect on the world, and what should we expect to see over the next couple of years.
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability.
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi’s VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context wi...
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of (at least) three separate application components: the software embedded in the device, the back-end service, and the mobile application for the end user’s controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target – this makes the integration of these separate pipelines and the coordination of software updates for...
In his session at @ThingsExpo, Ben Bromhead, CTO of Instaclustr, will walk you through the basics of building an IoT-based platform leveraging Cassandra, Spark and Kafka. This session is aimed at developers, admins and DevOps engineers who have to build, run and maintain high performance IoT platforms as well as data scientists/engineers who are sick of ETL and want to work with the most up to date information.
From Fitbits, to connected cars, to sensors that water crops, making decisions is no longer enough, now you need to make decisions in context. To bring your next great IoT decision to life you need to be able to extend and connect beyond your traditional enterprise. But with thousands if not millions of different devices all with their own technologies, standards and security how do you connect with the right ones effectively. And technology is only one aspect of the challenge; how do you create the right business model to drive value from your solution? An IoT solution that doesn’t connect to...
After more than five years of DevOps, definitions are evolving, boundaries are expanding, ‘unicorns’ are no longer rare, enterprises are on board, and pundits are moving on. Can we now look at an evolution of DevOps? Should we? Is the foundation of DevOps ‘done’, or is there still too much left to do? What is mature, and what is still missing? What does the next 5 years of DevOps look like?
In this Power Panel at DevOps Summit, moderated by DevOps Summit Conference Chair Andi Mann, panelists w...
Martin Fowler describes how infrastructure automation is a key enabler of microservices:
“Many of the products or systems being build with microservices are being built by teams with extensive experience of Continuous Delivery and it’s precursor, Continuous Integration. Teams building software this way make extensive use of infrastructure automation techniques. This is illustrated in the build pipeline shown below.”
IT is often cast in a dichotomy where things are either viewed as traditional, stable, and safe or innovative, impermanent, and vulnerable. There have been multiple attempts to organize these classifications in a fashion that make IT budget and planning more predictable such as various agile methodologies, DevOps, and the bimodal model.
What qualifies as success depends on the customer and producer involved in a project. A survey conducted by Scott Ambler in 2013 suggests that being on schedule...
Most everyone in Cloud IT circles has realized the power of containerization and that companies are adopting Docker containers at a remarkable rate. There are many good reasons for this, such as easily setting up dev/test scenarios (DevOps), and building out sophisticated, distributed computing clusters. But there are some deeper questions this talk will address from the Microsoft perspective. For example, what is the future of Windows in a containerized world? How will Windows and Linux work to...
Docker is hot. However, as Docker container use spreads into more mature production pipelines, there can be issues about control of Docker images to ensure they are production-ready. Is a promotion-based model appropriate to control and track the flow of Docker images from development to production?
In his session at DevOps Summit, Fred Simon, Co-founder and Chief Architect of JFrog, will demonstrate how to implement a promotion model for Docker images using a binary repository, and then show h...
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds.
How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem"...
Ever since the dawn of the Internet, people have struggled with how to get one computer to talk to another. Early business systems had no provision for such interactions. They were entirely closed— worlds unto themselves.
As enterprises set up early networks, the question of how to get applications to interact with each other became a pressing business concern, and led to the introduction of Remote Procedure Calls (RPCs). A client computer might directly interact with the program running on a...
No one should need to be convinced the value of good data. It gives you the confidence to make decisions quickly and with less risk, it allows you to measure your success, and it lets you know when you need to adjust your course. But there’s a difference between knowing the value of data, and creating a culture around it. A data-driven culture is a culture where everyone quantifies their actions as much as possible, and asks themselves how their teams are having a tangible impact on the business...
I have been following containers for quite some time now.
A year ago it was safe to say that container technologies like Docker were far from production ready when it came to security.
What I have seen over the past year is a ton of development towards closing that gap.
Check out my latest post on The Virtualization Practice where I dicuss how Docker is closing the gap on container security.
Most everyone in Cloud IT circles has realized the power of containerization and that companies are adopting Docker containers at a remarkable rate. There are many good reasons for this, such as easily setting up dev/test scenarios (DevOps), and building out sophisticated, distributed computing clusters. But there are some deeper questions this talk will address from the Microsoft perspective. For example, what is the future of Windows in a containerized world? How will Windows and Linux work to...
Test-Driven Development is a great tool for functional testing, but can you apply the same technique to performance testing? Why not? The purpose of TDD is to build out small unit tests, or scenarios, under which you control your initial coding. Your tests will fail the first time you run them because you haven’t actually developed any code. But once you do start coding, you’ll end up with just enough code to pass the test. There’s no reason the same philosophy can’t be applied to performance te...
What it means to build quality software has taken a beating over the years. We’re no longer content to strive for defect-free code. We must also make sure the software meets both its functional and nonfunctional requirements. Only now with the rise of more Agile ways of thinking, we’ve placed the notion of a software requirement under the microscope, as building flexible, resilient software often trumps checking items off our requirements list.
In today's digital world, change is the one constant. Disruptive innovations like cloud, mobility, social media, and the Internet of Things have reshaped the market and set new standards in customer expectations. To remain competitive, businesses must tap the potential of emerging technologies and markets through the rapid release of new products and services. However, the rigid and siloed structures of traditional IT platforms and processes are slowing them down – resulting in lengthy delivery ...
At some point you’ve probably heard the term “test early and often.” If you are in an Agile organization, that term perfectly captures the philosophy of iterative development and the commitment to rooting out defects sooner rather than later. It's nice - maybe even ironic - that a phrase which had such unscrupulous origins is now a hallmark characteristic of a process that exemplifies teamwork and quality. It's in that modern spirit that we wanted to share these 10 strategies to help you test ea...
Agile software development and agile marketing have followed similar, but unique paths. Buggy software, product delays and death marches were the norm in the 90s. A groundswell formed around the idea of taking the agile principles that had been so successful in manufacturing and applying them to software.
We’ve all had that moment where a friend comments (or complains) on social media about the new layout of a particular site. But you still see it the way it has been for months, and you have no idea what they are talking about. This is because the company that runs the site is likely using a Canary Release.
Containers are a hot topic these days. I have run a few workshops with clients, and one of the questions I get asked most frequently is “what are companies using containers for?”
After answering this question a number of times, I thought I would share some common use cases with my readers.
Check out my latest post at the Virtualization Practice where I highlight 3 common use cases.
As with most modernized economies, the United States economy utilizes capitalist principles. It is only fitting that we invented a technological solution that will help companies engage in c-api-talism using APIs in a more efficient manner.
If we look back into the history of mankind, we progressed towards more civilized, mature, monetary-based economies on a steady basis. The progression from the Stone Age, to the Bronze Age, to the Iron Age, to the Industrial Age and to the current Digital ...





















