Welcome!

Java Authors: Sharon Barkai, Adrian Bridgwater, Imran Akbar, Pat Romanski, Mat Mathews

Related Topics: DevOps Journal, Java, Linux, AJAX & REA, Web 2.0

DevOps Journal: Article

Software Quality Metrics For Continuous Delivery | @DevOpsSummit [#DevOps]

Even small changes need to be tracked and their impact on overall software quality must be measured

Software Quality Metrics for Your Continuous Delivery Pipeline | Part I

How often do you deploy new software? Once a month, once a week or every hour? The more often you deploy the smaller your changes will be. That's good! Why? Because smaller changes tend to be less risky since it's easier to keep track of what has really changed. For developers, it's certainly easier to fix something you worked on three days ago than something you wrote last summer. An analogy from a recent conference talk from AutoScout24 is to think about your release like a container ship, and every one of your changes is a container on that ship:

Your next software release en route to meet its iceberg

If all you know is that you have a problem in one of our containers you'd have to unpack and check all of them. That doesn't seem to make sense for a ship, and neither does it for a release. But that's still what happens quite frequently when a deployment fails and all you get is "it didn't work." In contrast, if you were shipping just a couple of containers you would be able to replace your giant, slow-maneuvering vessel with something faster and more agile - and if you're looking for a problem, you'd only have to inspect a handful of containers. While adopting this practice in the shipping industry would be a rather costly approach, this is exactly what continuous delivery allows us to do: Deploy more often, get faster feedback, and fix problems faster.

A great example is Amazon, who shared their success metrics at Velocity:

Some impressive stats from Amazon showing the success of rapid continuous delivery

However - even small changes can have severe impacts. Examples?

  1. Heavy DOM Manipulations through JavaScript: Introduced through a "harmless" new JavaScript library for tracking link clicks
  2. Memory Leaks in Production: Introduced by a not well tested remote logging framework downloaded on GitHub
  3. Performance Impact of Exceptions in Ops: Ops and Dev did not follow the same deployment steps (lack of automation scripts) resulting in thousands of exceptions and maxes out CPU on all app servers

Extending Your Delivery Pipeline
Even small changes need to be tracked and their impact on overall software quality must be measured along the delivery pipeline so that your quality gates can stop even the smallest change from causing a huge issue. The three examples above could have been avoided when automatically looking at the following measures across the delivery pipeline and stopping the delivery when "architectural" regressions are detected:

  • The number of DOM manipulations
  • Memory usage or object churn rate per transaction
  • Number of exceptions, number of database queries or number of log entries.

In a series of blog posts I will introduce you to metrics that you have to measure along your pipeline to act as an additional quality measure mechanism in order to prevent problems listed above. It is important that:

  • Developers get these measurements in the commit stage
  • Automation Engineers need to measure them for the automated unit and integration tests
  • Performance Engineers add them to the load testing reports you do in staging
  • Operations verify how the real application behaves after a new deployment in production

For each metric I introduce, I'll explain why it is important to monitor it, which types of problems can be detected and how Developers, Testers and Operations can monitor these metrics. To ready more on this, click here for the full article.

More Stories By Andreas Grabner

Andreas Grabner has more than a decade of experience as an architect and developer in the Java and .NET space. In his current role, Andi works as a Technology Strategist for Compuware and leads the Compuware APM Center of Excellence team. In his role he influences the Compuware APM product strategy and works closely with customers in implementing performance management solutions across the entire application lifecycle. He is a frequent speaker at technology conferences on performance and architecture-related topics, and regularly authors articles offering business and technology advice for Compuware’s About:Performance blog.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
The Industrial Internet of Things represents a tremendous opportunity for innovative companies looking to unlock new revenue sources by packaging their products with new digital services, says Accenture (NYSE:ACN) in its new report, “Driving Unconventional Growth through the Industrial Internet of Things.” Combining sensor-driven computing, industrial analytics and intelligent machine applications into a single universe of connected intelligent industrial products, processes and services, the Industrial Internet of Things generates data essential for developing corporate operational efficie...
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at Internet of @ThingsExpo, Erik Lagerway, Co-founder of Hookflash, will walk through the shifting landscape of traditional telephone and voice s...
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With “smart” appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user’s habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps, abiding by privacy concerns and making the concept a reality. These challenges can’t be addressed w...
The world's leading 'Internet of Things' event, @ThingsExpo has launched IoT Journal on the SYS-CON.com portal, featuring over 5,500 original articles, news stories, features, and blog entries. IoT Journal becomes the world's leading resource for the Internet of Things. SYS-CON Media CEO Carmen Gonzalez is founder and publisher of IoT Journal, and Roger Strukhoff, long-time SYS-CON editor and the conference chair of @ThingsExpo is the editor of the world's leading IoT resource. IoT Journal offers top articles, news stories, and blog posts from the world's well-known experts and guarantees bett...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world. The next @ThingsExpo will take place November 4-6, 2014, at the Santa Clara Convention Center, in Santa Clara, California. Since its launch in 2008, Cloud Expo TV commercials have been aired and CNBC, Fox News Network, and Bloomberg TV. Please enjoy our 2014 commercial.
The Internet of Things (IoT) is rapidly in the process of breaking from its heretofore relatively obscure enterprise applications (such as plant floor control and supply chain management) and going mainstream into the consumer space. More and more creative folks are interconnecting everyday products such as household items, mobile devices, appliances and cars, and unleashing new and imaginative scenarios. We are seeing a lot of excitement around applications in home automation, personal fitness, and in-car entertainment and this excitement will bleed into other areas. On the commercial side, m...
We were in contact recently with Shrikant Pattathil (pictured below), Executive Vice President of Harbinger Systems. Here are some of his thoughts about healthcare, the IoT, and disruption: IoT Journal: Healthcare, with all of its systems and dataflows, seems an ideal area for IoT solutions. What is Harbinger Systems doing in this area? Shrikant Pattathil: Being a service provider we work with many product development companies who are building new IoT-based applications to solve problems that plague the healthcare industry. For example, there is a need for applications to manage your medicin...
Where historically app development would require developers to manage device functionality, application environment and application logic, today new platforms are emerging that are IoT focused and arm developers with cloud based connectivity and communications, development, monitoring, management and analytics tools. In her session at Internet of @ThingsExpo, Seema Jethani, Director of Product Management at Basho Technologies, will explore how to rapidly prototype using IoT cloud platforms and choose the right platform to match application requirements, security and privacy needs, data managem...
Technology is enabling a new approach to collecting and using data. This approach, commonly referred to as the “Internet of Things” (IoT), enables businesses to use real-time data from all sorts of things including machines, devices and sensors to make better decisions, improve customer service, and lower the risk in the creation of new revenue opportunities. In his session at Internet of @ThingsExpo, Dave Wagstaff, Vice President and Chief Architect at BSQUARE Corporation, will discuss the real benefits to focus on, how to understand the requirements of a successful solution, the flow of data...
The worldwide cellular network will be the backbone of the future IoT, and the telecom industry is clamoring to get on board as more than just a data pipe. In his session at Internet of @ThingsExpo, Evan McGee, CTO of Ring Plus, Inc., to discuss what service operators can offer that would benefit IoT entrepreneurs, inventors, and consumers. Evan McGee is the CTO of RingPlus, a leading innovative U.S. MVNO and wireless enabler. His focus is on combining web technologies with traditional telecom to create a new breed of unified communication that is easily accessible to the general consumer. ...
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, will describe how to revoluti...
There are dozens of disruptive, innovative, truly ground-breaking connected devices on the market today. Most of them, however, have not achieved anything close to the kind of ubiquity that they are seeking. Why? Because in the wearable tech industry, innovation alone is not enough. In order to be adopted by mainstream audiences, a device must be both disruptive and unobtrusive – it must slip into our lives without us having to adjust our behavior, or even really think about its presence. In his session at Internet of @ThingsExpo, Gilles Bouchard, CEO of Livescribe, will discuss the role tha...
Building low cost wearable devices can enhance the quality of our lives. In his session at Internet of @ThingsExpo, Sai Yamanoor, Embedded Software Engineer at Altschool, will provide an example of putting together a small keychain within a $50 budget that educates the user about the air quality in their surroundings. He will also provide examples such as building a wearable device that provides transit or recreational information. He will review the resources available to build wearable devices at home including open source hardware, the raw materials required and the options available to pow...
Larry Ellison turned 70 and has decided to turn over the CEO reins at Oracle. Safra Catz and Mark Hurd, both in their 50s, will function as a “Ms. Inside and Mr. Outside” as co-CEOs, at least for awhile. Serious reverberations will be felt within this highly competitive company and the highly competitive industry in which it makes its money. Even while guiding his yacht to an America's Cup title, Larry Ellison remained in firm control of the company he founded in 1977. He still has an ownership stake of about 20% of the company--1 billion or so shares of Oracle stock worth about $40 bil...
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at Internet of @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, will show you how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
The Domain Name Service (DNS) is one of the most important components in networking infrastructure, enabling users and services to access applications by translating URLs (names) into IP addresses (numbers). Because every icon and URL and all embedded content on a website requires a DNS lookup loading complex sites necessitates hundreds of DNS queries. In addition, as more internet-enabled ‘Things’ get connected, people will rely on DNS to name and find their fridges, toasters and toilets. According to a recent IDG Research Services Survey this rate of traffic will only grow. What’s driving th...
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and assessments, including a decade of leading incident response and digital forensics. He is co-author of t...
It's time to condense all I've seen, heard, and learned about the IoT into a fun, easy-to-remember guide. Without further ado, here are Five (5) Things About the Internet of Things: 1. It's the end-state of Moore's Law. It's easy enough to debunk the IoT as “nothing new.” After all, we've have embedded systems for years. We've had devices connected to the Internet for decades; the very definition of a network means things are connected to it. But now that the invariable, self-fulfilling prophecy of Moore's Law has resulted in a rise from about 10,000 transistors on a chip in 1980 to more t...
As the Internet of Things gains momentum, the focus has been on securing billions of IoT devices and the servers that orchestrate their connectivity. However, the greatest security and authentication risks reside within the communications among devices and servers. In his session at Internet of @ThingsExpo, Todd Greene, Founder & CEO of PubNub, will discuss the top 10 challenges in securing IoT communications that, unsolved, render it impossible to deliver a secure IoT rollout. Learn the requirements for a ubiquitous, secure, bi-directional communication protocol for IoT. Specific design patt...
It's the Great Convergence! That is, the convergence of the IoT and WebRTC. “From telemedicine to smart cars, digital homes and industrial monitoring, the explosive growth of IoT has created exciting new business opportunities for WebRTC, real time calls and messaging,” says Ivelin Ivanov, CEO and Co-Founder of Telestar. Ivelin will be one of the featured speakers at our @WebRTCSummit, to be held Nov 4-5 as part of the overall @CloudExpo @ThingsExpo conference and exhibition Nov 4-6, at the Santa Clara Convention Center, Santa Clara, CA. In his session, Ivelin promises to share “some of the...