Consumer IoT applications provide data about the user that just doesn’t exist in traditional PC or mobile web applications. This rich data, or “context,” enables the highly personalized consumer experiences that characterize many consumer IoT apps. This same data is also providing brands with unprecedented insight into how their connected products are being used, while, at the same time, powering highly targeted engagement and marketing opportunities.
In his session at @ThingsExpo, Nathan Trel...| By AppDynamics Blog | Article Rating: |
|
| September 27, 2015 11:45 PM EDT | Reads: |
2,115 |

By John Rakowski
The next letter in the DevOps CALMS model brings us to M for Measurement. Focusing on the right metrics and measurements is vital if you are going to succeed with DevOps adoption.
“You Never Know Where You Are Going Until You Know Where You Have Been”
There are not many times in my career that I am going to be able to quote Will Smith, but in the “Will2K” video, Will states “You Never Know Where You Are Going Until You Know Where You Have Been”. For me, this summarizes why measurements and metrics are so important. Unless you can understand where you are right now then there is no way to understand whether you have been successful in regards to an initiative, in this case, DevOps adoption. Choosing the right DevOps measurements is important to eliciting the right behavior amongst IT teams, ensuring that they know what they need to focus on and, key to making sure that you are delivering the right value to the business.
Eight Essential DevOps Measurements
There are actually more than eight measurements that I would recommend for DevOps adoption. This is because your full measurement strategy really depends on what your stakeholders want to see, your goals and timescales for DevOps adoption. But at a top level, I believe there are eight metrics, which are essential, grouped around people, process and technology.
People measurements
In my experience most measurement strategies in regards to IT based initiatives focus on technology, but it’s ultimately your focus on people that will lead to success. This means employees and the customer. Two key metrics here are:
- Key employee retention – There will be employees in your team and your business that are key to DevOps success. Those who understand that DevOps is about taking an outside-in approach, focusing on business outcomes and delighting customers. Also those who can combine Development and Operations skills. These are skills you will want to hire, develop and retain. DevOps is a hot area from a job market perspective so you will want to make sure you keep talent in your business. Therefore identifying those employees who are vital to your DevOps efforts and then tracking retention is a must.
- Customer satisfaction/experience – Ultimately DevOps success is not about fast release, collaboration or automation etc. Rather it’s about making sure that software fueled business services satisfy and delight customers. This means that tying DevOps to customer satisfaction or experience measurements is vital. One option here is to link your DevOps measurements with NetPromoter scores if your company uses this popular customer satisfaction metric.
Process measurements
From a process perspective, the essential metrics which focus on release and change are:
- Release/change frequency – One of the key drivers for DevOps adoption is to release new features or applications quicker than before. The aims here are to make sure that your application users (customers or employees) receive the value they need, are happy with functionality and ultimately stay loyal to your business. So measuring release/change frequency is essential. It could also be said that release/change frequency is one lagging metric for the leading measurement of customer satisfaction/experience.
- Volume of failures – This measurement goes hand-in-hand with the measurement above. There is no point being able to release quickly if these releases fail and have to be rolled back.
- Time/cost per release – Ok, maybe this should technically be two metrics. Time is money and money is required for a successful business. This means that it’s critical to measure the time taken and cost per release especially in regards to Ecommerce based applications. If the cost of frequent releases outweighs the profit brought in by the application, then your DevOps adoption is not generating the business benefits that it needs to.
Technology measurements
There are a number of technology metrics that you may choose for your DevOps adoption but here are my three essentials:
- Mean Time To Resolution (MTTR) – As mentioned, ultimately DevOps is about making sure that software strategy drives successful business outcomes by providing applications that help to deliver the service customers want. As such performance or availability issues are a DevOps strategy killer, and focusing on keeping MTTR measurements as low as possible is important. To improve, an end-to-end monitoring solution, which monitors end-user experience through to the application database or data store is critical.
- Mean Time Between Failure (MTBF) - This metric comes from software engineering practices and focuses on making sure that applications are available. In the software-defined business highly available applications are key as they fuel business success. Therefore this metric is important BUT
- Mean Time To Customer Impact – The reality is that failure only becomes an issue when it impacts a business process or the customer. Also failure as per my culture blog post is not a negative in DevOps as the mantra should be fail-fast, fail-forward. A good APM platform will help you to isolate and automatically remediate emerging application issues quickly before they impact the customer. Therefore focusing on Mean Time To Customer Impact is an essential metric for DevOps and monitoring efficiency.
So here are my essential DevOps metrics. As mentioned, the list is not exhaustive but critical to any DevOps strategy. As always comments and feedback is welcome.
If you want to see how AppDynamics can support your DevOps adoption then please try our platform for free
The post Keep CALM And Embrace DevOps – Measurement appeared first on Application Performance Monitoring Blog | AppDynamics.
Published September 27, 2015 Reads 2,115
Copyright © 2015 SYS-CON Media, Inc. — All Rights Reserved.
Syndicated stories and blog feeds, all rights reserved by the author.
More Stories By AppDynamics Blog
In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.
DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.
Consumer IoT applications provide data about the user that just doesn’t exist in traditional PC or mobile web applications. This rich data, or “context,” enables the highly personalized consumer experiences that characterize many consumer IoT apps. This same data is also providing brands with unprecedented insight into how their connected products are being used, while, at the same time, powering highly targeted engagement and marketing opportunities.
In his session at @ThingsExpo, Nathan Trel...Jan. 19, 2016 07:00 AM EST Reads: 1,004 |
By Pat Romanski Who are you? How do you introduce yourself? Do you use a name, or do you greet a friend by the last four digits of his social security number? Assuming you don’t, why are we content to associate our identity with 10 random digits assigned by our phone company? Identity is an issue that affects everyone, but as individuals we don’t spend a lot of time thinking about it.
In his session at @ThingsExpo, Ben Klang, Founder & President of Mojo Lingo, discussed the impact of technology on identity. Sh...Jan. 19, 2016 06:15 AM EST Reads: 387 |
By Elizabeth White Most people haven’t heard the word, “gamification,” even though they probably, and perhaps unwittingly, participate in it every day.
Gamification is “the process of adding games or game-like elements to something (as a task) so as to encourage participation.” Further, gamification is about bringing game mechanics – rules, constructs, processes, and methods – into the real world in an effort to engage people.
In his session at @ThingsExpo, Robert Endo, owner and engagement manager of Intrepid D...Jan. 19, 2016 05:00 AM EST Reads: 124 |
By Liz McMillan Learn how IoT, cloud, social networks and last but not least, humans, can be integrated into a seamless integration of cooperative organisms both cybernetic and biological. This has been enabled by recent advances in IoT device capabilities, messaging frameworks, presence and collaboration services, where devices can share information and make independent and human assisted decisions based upon social status from other entities.
In his session at @ThingsExpo, Michael Heydt, founder of Seamless...Jan. 19, 2016 05:00 AM EST Reads: 537 |
By Pat Romanski Jan. 19, 2016 02:45 AM EST Reads: 270 |
By Pat Romanski Contrary to mainstream media attention, the multiple possibilities of how consumer IoT will transform our everyday lives aren’t the only angle of this headline-gaining trend. There’s a huge opportunity for “industrial IoT” and “Smart Cities” to impact the world in the same capacity – especially during critical situations. For example, a community water dam that needs to release water can leverage embedded critical communications logic to alert the appropriate individuals, on the right device, as...Jan. 19, 2016 02:00 AM EST Reads: 767 |
By Liz McMillan The Internet of Things has the potential to disrupt all industries, not just consumer, as businesses leverage the new insights and capabilities enabled by new devices / things, automation, integration and analytics, etc., to transform how they do business.
One industry ripe for disruption is higher education. Colleges and universities are being challenged with serving more students and at the same time ensuring successful student outcomes.
In his session at @ThingsExpo, Chris Witeck, Principa...Jan. 19, 2016 01:15 AM EST Reads: 585 |
By Elizabeth White The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today. Jan. 18, 2016 10:45 PM EST Reads: 838 |
By Elizabeth White For manufacturers, the Internet of Things (IoT) represents a jumping-off point for innovation, jobs, and revenue creation. But to adequately seize the opportunity, manufacturers must design devices that are interconnected, can continually sense their environment and process huge amounts of data.
As a first step, manufacturers must embrace a new product development ecosystem in order to support these products. Jan. 18, 2016 05:45 PM EST Reads: 308 |
By Liz McMillan Jan. 18, 2016 05:30 PM EST Reads: 349 |
By Liz McMillan WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was.
You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.Jan. 18, 2016 03:00 PM EST Reads: 313 |
By Elizabeth White With the exponential growth of network traffic slowing down data transmission, companies are looking for solutions. Recently, a solution has emerged that can help improve your data speed with data centers on the edge. These micro data center solutions can simplify the lives of many data center owners and operators because they are self-contained, secure computing environments, assembled in a factory and shipped in one enclosure which includes all the necessary power, cooling, security, and manag...Jan. 18, 2016 03:00 PM EST Reads: 540 |
By Elizabeth White Jan. 18, 2016 03:00 PM EST Reads: 378 |
By Carmen Gonzalez NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...Jan. 18, 2016 02:30 PM EST Reads: 464 |
By Elizabeth White Jan. 18, 2016 01:15 PM EST Reads: 379 |
By Elizabeth White The Internet of Things is in the early stages of mainstream deployment but it promises to unlock value and rapidly transform how organizations manage, operationalize, and monetize their assets. IoT is a complex structure of hardware, sensors, applications, analytics and devices that need to be able to communicate geographically and across all functions. Once the data is collected from numerous endpoints, the challenge then becomes converting it into actionable insight.Jan. 18, 2016 01:00 PM EST Reads: 467 |
By Elizabeth White When it comes to IoT in the enterprise, namely the commercial building and hospitality markets, a benefit not getting the attention it deserves is energy efficiency, and IoT’s direct impact on a cleaner, greener environment when installed in smart buildings. Until now clean technology was offered piecemeal and led with point solutions that require significant systems integration to orchestrate and deploy. There didn't exist a 'top down' approach that can manage and monitor the way a Smart Buildi...Jan. 18, 2016 01:00 PM EST Reads: 749 |
By Elizabeth White As the Internet of Things becomes more capable, moving from an initial ad hoc bundle of chaos to an optimized set of interconnected assets, we will need to ensure it performs and is reliable.
But this means the tools measuring the performance and reliability of the IoT will also need to mature as the need for timely, understandable, quality, business-critical information becomes paramount. Sensors that are malfunctioning, devices that are offline, data that’s little more than noise can all scut...Jan. 18, 2016 12:30 PM EST Reads: 350 |
By Elizabeth White Contextual Analytics of various threat data provides a deeper understanding of a given threat and enables identification of unknown threat vectors.
In his session at @ThingsExpo, David Dufour, Head of Security Architecture, IoT, Webroot, Inc., discussed how through the use of Big Data analytics and deep data correlation across different threat types, it is possible to gain a better understanding of where, how and to what level of danger a malicious actor poses to an organization, and to determ...Jan. 18, 2016 12:15 PM EST Reads: 818 |
By Liz McMillan SYS-CON Events announced today that Interface Masters Technologies, a leading vendor in the network monitoring and high speed networking markets, will exhibit at the 18th International CloudExpo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.Jan. 18, 2016 12:15 PM EST Reads: 372 |

Who are you? How do you introduce yourself? Do you use a name, or do you greet a friend by the last four digits of his social security number? Assuming you don’t, why are we content to associate our identity with 10 random digits assigned by our phone company? Identity is an issue that affects everyone, but as individuals we don’t spend a lot of time thinking about it.
In his session at @ThingsExpo, Ben Klang, Founder & President of Mojo Lingo, discussed the impact of technology on identity. Sh...
Most people haven’t heard the word, “gamification,” even though they probably, and perhaps unwittingly, participate in it every day.
Gamification is “the process of adding games or game-like elements to something (as a task) so as to encourage participation.” Further, gamification is about bringing game mechanics – rules, constructs, processes, and methods – into the real world in an effort to engage people.
In his session at @ThingsExpo, Robert Endo, owner and engagement manager of Intrepid D...
Learn how IoT, cloud, social networks and last but not least, humans, can be integrated into a seamless integration of cooperative organisms both cybernetic and biological. This has been enabled by recent advances in IoT device capabilities, messaging frameworks, presence and collaboration services, where devices can share information and make independent and human assisted decisions based upon social status from other entities.
In his session at @ThingsExpo, Michael Heydt, founder of Seamless...
Contrary to mainstream media attention, the multiple possibilities of how consumer IoT will transform our everyday lives aren’t the only angle of this headline-gaining trend. There’s a huge opportunity for “industrial IoT” and “Smart Cities” to impact the world in the same capacity – especially during critical situations. For example, a community water dam that needs to release water can leverage embedded critical communications logic to alert the appropriate individuals, on the right device, as...
The Internet of Things has the potential to disrupt all industries, not just consumer, as businesses leverage the new insights and capabilities enabled by new devices / things, automation, integration and analytics, etc., to transform how they do business.
One industry ripe for disruption is higher education. Colleges and universities are being challenged with serving more students and at the same time ensuring successful student outcomes.
In his session at @ThingsExpo, Chris Witeck, Principa...
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
For manufacturers, the Internet of Things (IoT) represents a jumping-off point for innovation, jobs, and revenue creation. But to adequately seize the opportunity, manufacturers must design devices that are interconnected, can continually sense their environment and process huge amounts of data.
As a first step, manufacturers must embrace a new product development ecosystem in order to support these products.
WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was.
You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
With the exponential growth of network traffic slowing down data transmission, companies are looking for solutions. Recently, a solution has emerged that can help improve your data speed with data centers on the edge. These micro data center solutions can simplify the lives of many data center owners and operators because they are self-contained, secure computing environments, assembled in a factory and shipped in one enclosure which includes all the necessary power, cooling, security, and manag...
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
The Internet of Things is in the early stages of mainstream deployment but it promises to unlock value and rapidly transform how organizations manage, operationalize, and monetize their assets. IoT is a complex structure of hardware, sensors, applications, analytics and devices that need to be able to communicate geographically and across all functions. Once the data is collected from numerous endpoints, the challenge then becomes converting it into actionable insight.
When it comes to IoT in the enterprise, namely the commercial building and hospitality markets, a benefit not getting the attention it deserves is energy efficiency, and IoT’s direct impact on a cleaner, greener environment when installed in smart buildings. Until now clean technology was offered piecemeal and led with point solutions that require significant systems integration to orchestrate and deploy. There didn't exist a 'top down' approach that can manage and monitor the way a Smart Buildi...
As the Internet of Things becomes more capable, moving from an initial ad hoc bundle of chaos to an optimized set of interconnected assets, we will need to ensure it performs and is reliable.
But this means the tools measuring the performance and reliability of the IoT will also need to mature as the need for timely, understandable, quality, business-critical information becomes paramount. Sensors that are malfunctioning, devices that are offline, data that’s little more than noise can all scut...
Contextual Analytics of various threat data provides a deeper understanding of a given threat and enables identification of unknown threat vectors.
In his session at @ThingsExpo, David Dufour, Head of Security Architecture, IoT, Webroot, Inc., discussed how through the use of Big Data analytics and deep data correlation across different threat types, it is possible to gain a better understanding of where, how and to what level of danger a malicious actor poses to an organization, and to determ...
SYS-CON Events announced today that Interface Masters Technologies, a leading vendor in the network monitoring and high speed networking markets, will exhibit at the 18th International CloudExpo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
I recently spotted a five-year-old blog post by Mike Gualtieri of Forrester, where he suggests firing your quality assurance (QA) team to improve your quality. He got the idea from a client who actually tried and succeeded with this counterintuitive move.
The thinking goes that without a QA team to cover for them, developers are more likely to take care of quality properly – or risk getting the dreaded Sunday morning wakeup call to fix something.
Gualtieri’s post generated modest buzz at th...
Agile, which started in the development organization, has gradually expanded into other areas downstream - namely IT and Operations. Teams – then teams of teams – have streamlined processes, improved feedback loops and driven a much faster pace into IT departments which have had profound effects on the entire organization.
In his session at DevOps Summit, Anders Wallgren, Chief Technology Officer of Electric Cloud, discussed how DevOps and Continuous Delivery have emerged to help connect develo...
Web performance issues and advances have been gaining a stronger presence in the headlines as people are becoming more aware of its impact on virtually every business, and 2015 was no exception. We saw a myriad of major outages this year hit some of the biggest corporations, as well as some technology integrations and other news that we IT Ops aficionados find very exciting.
This past year has offered several opportunities for growth and evolution in the performance realm — even the worst failu...
Most of the Java developers I know don’t like JavaScript. Initially. They would give you different reasons why, but the real one is simple: too much to learn to make it work. For many Java developers creating the front end of a Web application in JavaScript is a chore to write and a burden to maintain. Nevertheless JavaScript rules in Web development and the new version of JavaScript (ES6) will make it even more popular.
Explore the 10 most popular continuous testing resources of 2015, including Forrester and Gartner research, interviews, infographics, an ebook, and more.
With today's DevOps and "Continuous Everything" initiatives, teams need the ability to assess the risks associated with a release candidate-instantly and continuously. Maybe that's why 2015 seemed to be all about Continuous Testing, from conferences, to analyst research, to the blogosphere.
Between the compelling mockups and specs produced by analysts, and resulting applications built by developers, there exists a gulf where projects fail, costs spiral, and applications disappoint. Methodologies like Agile attempt to address this with intensified communication, with partial success but many limitations.
In his session at DevOps Summit, Charles Kendrick, CTO and Chief Architect at Isomorphic Software, presented a revolutionary model enabled by new technologies.
As 2016 has arrived, we reflect upon one of the most debated issues around DevOps in 2015 – Information Security (InfoSec) and compliance. Needless to say, both are critical to an enterprise (especially given past examples of data breaches and looming cybersecurity threats). As a result, the combination of InfoSec and DevOps practices can be viewed as counter-intuitive, since the ability to “go faster” can be seen as a potential risk to security mechanisms in place, and thus harder to ensure com...
The web community is constantly churning out new technologies that make it easier to develop the applications that savvy users are demanding. It may be obvious, but new technologies often require new ways of thinking about testing – sometimes new tools, other times entirely new testing methods. You just can’t rely on the same old same old when things change as much as the web does.
Welcome to the first top DevOps news roundup of 2016! At the end of last year, we saw some great predictions for 2016.
While we’re excited to kick off the new year, this week’s top posts reminded us to take a second to slow down and really understand the current state of affairs. For example, do you actually know what microservices are – or aren’t? What about DevOps? Does the emphasis still fall mostly on the development side? This week’s top news definitely got the wheels turning and just migh...
According to the results of a new global study, commissioned by CA Technologies, 72% of organizations have implemented some aspect of DevOps but when the study took a closer look at the numbers, only 20% of DevOps users have put all the parts in place to reap the full benefits of DevOps.
Unlike many IT-related concepts, DevOps doesn't revolve around a specific type of technology, and it can't be classed as a methodology either. Indeed, DevOps generally requires blending a number of different te...
Today on #c9d9 we talked about best practices for orchestrating enterprise software testing. Delivering software faster, but with lower quality, doesn’t help you win in the market.
What are some of the challenges and proven tips and patterns used to test critical software – across all the different supported platforms – quickly and efficiently?
We started our Continuous Discussions (#c9d9) video podcast in late 2014 as a way for the DevOps community to connect with each others – sharing experiences, lessons learned and tips – to accelerate our mutual success.Each #c9d9 episode focuses on a different topic relating to your software delivery process, and features different speakers from around the development and IT community. Every two weeks, we get together for a casual video chat (over Hangout) to share best practices – and war storie...
Are you someone who knows that the number one rule in DevOps is “Don’t Panic”? Especially when it comes to making Continuous Delivery changes inside your organization? Are you someone that theorizes that if anyone implements real automation changes, the solution will instantly become antiquated and be replaced by something even more bizarre and inexplicable?
Fujitsu and Grid Dynamics have teamed up to create the Cloud Commerce Platform to empower retailers with best-of-breed hardware, software and services for faster digital innovation.
Retailers are under immense pressure to innovate faster. Working with them over the years, we have come to intimately understand their challenges. With the explosion of digital touch points and the expectation of a unified experience throughout a shopper's journey across the channels, retailers must innovate at an u...
When it comes to protecting their brand, there is little that any successful company would not do. But what if the brand is being damaged just by people being unable to find the store?
If you think of the internet as a virtual network, then DNS lookup and resolution is like the 411 service. Websites are not located at text addresses, but rather IP addresses – that series of numbers and decimals that may not make any sense to you, but actually identifies the actual server that you’re trying to...
With another year coming to a close, it’s time to dust off the crystal ball for our annual predictions. However, before we forecast any new insights, I want to look back at our 2015 predictions, which could be best summarized as a maturing of technology innovations introduced in prior years. Now that the fundamentals are in place, there’s potential for significant shifts in both software and vendor shakeups in 2016. The emerging battleground of innovation continues to heat up between companies t...
Ten years ago, when I drove on Winchester Avenue in Los Gatos, I saw this new company called Netflix that was renting DVDs of movies. You didn't have to go to a store, but could sign up for home delivery service of DVDs. Now Netflix is building a much bigger headquarters near its old site. It is a huge success story – a disruption to the home entertainment business. When it decided to switch to “streaming only” a few years back, there were lot of doubters. But that’s all in the past. It is a glo...
SDN / NFV may not be new with next gen telcos and data centers but by 2016 it is certainly going to be the new beast in DevOps space. It's a multi-million dollar virgin market, so it's not only those traditional OEMs are in the fray even the DB companies will be entering this new market. Telcos and Data center wants a vendor agnostic network devices and are expected to replace all physical devices including those routers and switches with software devices. This not only drastically brings down t...
If a slow or unresponsive application on your mobile device or PC has ever left you feeling angry, frustrated and unproductive, you're not alone. Business success in today's digital age relies on application performance. Yet according to the Riverbed Global Application Performance Survey 2015, nine out of 10 organizations suffer from poor performance on a regular basis. Executives say the poor performance of enterprise applications has negatively impacted their work on a regular basis. Consequen...


























