Storage virtualization for the sake of storage virtualization isn't enough these days.

Being able to pool heterogeneous resources and migrate data from point A to point B while the application is up and running is pretty cool, but what businesses really need are complete solutions - solutions that not only provision storage more efficiently, but can virtualize, protect, migrate, dedupe, encrypt, replicate, recover, and archive any data source in real-time, via policy, across both physical and virtual servers.

I've been running across more and more enterprises that are looking to implement a simple yet comprehensive solution that enables better efficiency in IT investments but saves costs by leveraging existing assets, reuses current policies and procedures, and reduces complexity by minimizing the number of components and management interfaces. What this entails is building an optimized suite of integrated data services on a common platform.

I call this holistic approach the "optimized data services" utility, or ODS model (see Figure 1). An ODS model is created by virtualizing existing data sets, storage, and servers to enable physical abstraction and flexible data movement between compute and storage elements.

Once virtualized, the ODS platform should allow the creation of policies that enforce specific service levels for explicit or pooled datasets. The grouping of data elements for consistency or recovery purposes shouldn't be hampered by physical constraints such as volumes in the same array, or SAN versus non-SAN, or network-attached devices or hosts. The engine should provide application-level interfaces so all data services can be provided from the aspect of the applications in question.

Simplicity of implementation is why the ODS solution needs to provide virtualization functions on existing LUNS that are already zoned through the storage fabric to existing servers. To simplify implementation of the ODS model into existing environments, the solution needs to provide the ability of bringing in existing data assets without having to migrate from its original location.

Most existing methods for virtualizing storage are based on "extent mapping" multiple physical disks into virtual disks into a pool of virtual storage, and then slicing the virtualized pool of physical disks into multiple vdisks (virtual disks) that can be granularly assigned to hosts (see Figure 2).

To bring existing data assets into these new virtual pools, all the existing data needs to be migrated into one of these new virtual pools, which can be time consuming and extremely disruptive. A fallback methodology would also need to be planned out in case issues arose or the solution failed.

Service enabling is much different than extent mapping, because it has the ability to add existing LUNS to a virtual storage infrastructure without migrating the data off existing LUNS (see Figure 3). Service enabling simply abstracts the physical LUN into a virtual LUN without touching the original data, and can even preserve the SCSI inquiry string for the LUN so the server still thinks it sees its original storage, even though the LUN has been virtualized. The host would still be able to use the original path failover software and volume management software.

Once a LUN is service-enabled through the ODS platform, all the higher-level functions of the ODS utility is immediately available to the LUN.

The ability of the solution to "service enable" existing Fibre Channel LUNS while preserving the SCSI inquiry string ID of each LUN enables almost transparent implementation, because nothing would really change on the hosts except for the WWN target of the storage ports. This is much different than virtualization through storage pooling, since the original data is untouched, and not migrated. The ODS solution can be implemented via a simple zone change in the fabric to include the virtualization layer appliances in the data path. Once the abstraction layer is in place, all writes to the original LUNS can be protected, migrated, or replicated for recovery, and the data in the existing LUN could then be transparently migrated in real-time into a true virtualized pool that matches the requirements of the application. (Virtual pool properties would be based on reliability, cost, performance, and other factors.)

The ODS solution engine should be able to provide thin provisioning capabilities for enhancing storage utilization; and capacity expansion for running applications should occur in real-time and on-demand by the compute resources in question. This reduces the overall administrative burden and provides an element of automation to the design. All data should be continually protected based on policy and recovery time objectives (RTO) and recovery point objectives (RPO) should be reachable at minimal costs based not on budget constraints, but by the service level agreement (SLA) policy applied to the application. This unique capability can only be achieved if the solution also automatically applies efficiency in data storage and movement through the de-duplication and sub-block-level monitoring of all stored data to ensure only unique data is stored and replicated.

A focus on application uptime and rapid recovery is paramount in such a design, so the solution must also be able to integrate at the application level and provide a simple means to monitor and recover any application across any platform from hardware and software failure or malicious intent. Protection from corruption and deletion is also very important, so continuous protection needs to be utilized to achieve a zero RPO for critical applications. Using such a solution for critical applications eliminates the need for multiple management elements for protection and replication, such as log shipping and array-based replication. Also, since protection is continuous and policy-based, there's no requirement for backup applications, clients, servers, media, or processes, which saves huge sums of money and time and allows companies to focus on the business rather than IT technologies. The engine must also be intelligent and work seamlessly with or even enhance other protection and virtualization solutions such as VMware and VMware Site Recovery Manager, Microsoft Failover Clustering and Data Protection Manager, Oracle Real Application Clusters, SAP BRtools, PolyServe, Platform Computing, Virtual Iron, Citrix, Sybase Replication Server, and others.

A comprehensive ODS utility would need to provide built-in encryption and off-site replication of all data sets for risk mitigation. To reduce WAN costs, data optimization over WAN links needs to be included. Since many organizations are obliged by law or regulation to provide removable media copies, the ability to transparently integrate with tape formats and tape-based archiving is beneficial. Since tape is also low-cost and removable, it should be used for long-term archives and data should move transparently based on policy to tape-based media. The media should be automatically encrypted by the solution and not require expensive tape hardware or libraries that enable encryption. Furthermore, data must be stored in an immutable fashion for compliance, and it must be searchable for audit purposes. All datasets also should be de-duplicated so that only a single instance of every data object is archived.

Data de-duplication, however, should not be used when storing data in native format ready for application use or when rapid recovery (under a couple of minutes) is required. Since data de-duplication usually implies electronic hashing of data into unique objects, a recovery process would need to be applied to reconstitute the data. Instead, data should simply be stored more efficiently by monitoring the data stream and eliminating any "white space" in the file system or data blocks written by the application. During data replication, only these unique sectors of disk would have to be replicated and stored for recovery at the disaster recovery (DR) site. By simply storing data more efficiently, companies gain the benefits of data de-duplication without the associated overhead or risk, and the datasets themselves are always instantly available for mounting to the same or a different application for recovery, testing, or DR. In fact, if data can be stored very efficiently, these space-efficient images can be used for retaining multiple data points for many days, providing the ability to recover applications very rapidly to any point in time while saving costs.

The ODS utility should be flexible enough to accommodate not only existing protocols such as Fibre Channel and iSCSI but also newer protocols such as Fibre Channel over Ethernet (FCoE) and Infiniband, so that rapid obsolescence can be avoided. The refresh of any component should be transparent to running applications and maintenance must be done with minimal or no downtime. Scalability should be a simple factor of adding more compute resources, connections, or ports in a modular fashion and not limited or hampered by technical issues or artificial resource limitations of the file system, capacity, connectivity, or availability.

The ODS utility would be more cost-effective if it could provide these capabilities using the same server and storage infrastructure currently in place. There would be no need to buy proprietary disks or servers to create the solution. The accumulated knowledge of the existing environment wouldn't be wasted and the learning curve would be greatly reduced.

Recovery from failure or disaster should be simple, fast, comprehensive, cost-effective, and, where possible, provide automation capabilities. At the very least, recovery should be simple enough so that at time of failure no one has to scramble to figure out how recovery actually works. This means that the ability to test for DR should be intrinsic in the design and simplified to the point that following a wizard or script is all operations staff should need to know. Since many applications also include data feeds from other applications, the ability to provide consistency grouping for recovery across platforms and storage tiers is also a requirement for the ODS utility.

This is a pretty tall order; but, once achieved, it would make it easy for IT staff to stay home with family or friends, drinking their favorite alcoholic beverage, rather than coming into work at 3:00 a.m. to recover someone's mistake. That said, the ODS utility should be able to be implemented intuitively and rapidly without requiring weeks or months of professional services to make it work.

In fact, it would be very cool if all you needed to do to create a node that operated in the ODS platform was to take an existing server, grab a USB memory stick that includes all the self-installable software you need, put the stick in an open USB port, and reboot the server. Do that to as many servers as you need for the required performance and you could build your own platform for optimized data services - or PODS - which would provide the critical data services and abstraction. Simply attach a POD to your storage network, zone it in, and you're done. Attach two PODS together across the WAN at two sites, and you have DR - while only unique and encrypted data traverses the connection between the PODS. You could even create "mini-PODS" for your remote locations by using small low-cost servers with internal storage or install the solution on a VMware virtual server and do it all virtually!

Companies looking to optimize their data services and create a more service-oriented architecture for their applications and data resources or looking at moving to a cloud computing model should take a hard critical look at solutions currently available in the market. You don't want to have to tie together software from multiple vendors into a Frankenstein-like science project that would be a support nightmare. Be sure to look for a platform that provides all the capabilities I've mentioned, so you can implement simply, quickly, and with peace of mind, knowing that everything is certified, supportable, and manageable globally from a single console.