Customers often ask us whether archiving or partitioning is the best practice for ILM. This question usually is based on a misunderstanding, because both can be used to address performance and storage cost issues. Partitioning and archiving is not an either/or choice. Rather they are complimentary, as evidenced by this from Gartner. So the correct answer to this question is yes, and Solix EDMS supports partitioning, archiving, and retiring data to the cloud for all your applications across your enterprise.

According to Orafaq, database partitioning is an option that database companies sell on top of a database license . Partitioning allows DBAs to split large tables into more manageable "sub-tables", called partitions, to improve database performance, manageability, and availability. It supports scaling of certain applications and provides the ability to leverage tiered storage to take advantage of lower-cost storage options. Whether partitioning is a good data management strategy depends in large part on how the individual application is architected. For instance, d ata warehouse applications are generally good partitioning candidates because date is usually a key dimension and can be used to spread the data out evenly (by years, quarters, months, etc.) This segregation can be used to improve query performance by excluding ‘offline’ partitions that are no longer needed for day-to-day operations. If applications are not architected with partition elimination strategy in their queries  it doesn’t significantly improve database performance as partitioned data still resides in the database as against moving complete sets of data out of the database does.

Database archiving, on the other hand, is the process of moving data that is no longer actively used to a separate data storage device such as SATA disk or tape, for long-term retention. This strategy is recommended for older data no longer active in the production environment but important for reference or regulatory compliance. Generally, this is achieved by creating two versions of each table, one for active data, and one for archived data. However, this means that all queries that need to access the archived data must now select a union of the active and archive tables. Database archiving products manage this data movement in an optimized manner while maintaining the integrity of the application.

Oracle, in its white paper “ Using Database Partitioning with Oracle E-Business Suite", for instance, recommends partitioning for improving performance and saving cost and reducing maintenance activities by moving old partitions to read-only table spaces (pg 49). The biggest problem when defining partition strategies for the Oracle E-Business Suite is that the application code has been prebuilt, which limits the choice of keys. Furthermore, changes to the partitioning strategies delivered with the base product are not recommended or supported. Therefore, the introduction of partitioning is complex, requiring both substantial analysis and robust testing of its effect on different components in your workload. This makes archiving of older data more attractive. A lso, by itself database partitioning does not meet long-term archiving and compliance requirements. On the other hand, transaction-based archiving alone may not be able to keep up with the volume demands of certain applications. Together they can provide a highly complementary solution that maintains performance of the production database while saving money.

Solix EDMS is designed to help with your partitioning and archiving strategy, as Gartner acknowledges in reports such as “ Solid Vendor Solutions Bolster Database Archiving Market ” by Sheila Childs and Alan Dayley. In that report, Solix is listed along the four important vendors for database archiving. Solix EDMS can be used to apply a consistent set of company ILM policies for data archiving and data retirement across all your data and applications. And it can provide a solution for retaining read-only access to data when retiring obsolescent applications, an important strategy for data center efficiency that is strongly supported in the Gartner report cited above.

On another note, I am happy to announce that we will co-sponsor www.ilmsolutions.net . This is an excellent resource for end-users researching ILM strategy, as it publishes all the information on the market, including industry white papers, analyst reports and customer case studies.

As CPU power continues to grow according to Moore’s Law and increasing numbers of customers see the advantages of near real-time data access, big data ingestion and predictive analysis and similar advanced computing strategies, data read/write speeds have become a major barrier to achieving high performance. The basic problem is that disk spins at the same speed that it did 30 years ago, which means disk reads and writes, particularly random accesses, can only happen so fast. And the steady increase in disk capacity has, if anything exacerbated rather than ameliorated this problem.

In the last two years, however, a new technological answer has entered the market – NAND flash storage providing a 20X to 40X performance improvement on PCIe cards plugged directly into the server. While initial NAND flash implementations were very expensive and exotic, their price has been falling steadily due mainly to the heavy demand from the consumer marketplace. Flash SSDs are highly reliable and have much lower power consumption and therefore cooling costs than disk storage, in part because they only need power when they are actually in use. For these reasons, NAND Flash technology is becoming a disruptive force in the storage hardware industry. Several vendors are offering flash in different configurations at different price points, and flash storage has become a practical and much higher performance, alternative to high-speed disk for data-intensive, business-critical applications such as ERP and CRM systems even for SMBs.

NAND flash, however, is significantly more expensive than SATA disks on a per Gbyte basis. Hence, the optimal way of using flash is to replace high-speed disk entirely for the most frequently accessed Tier 0 and Tier 1 data in business-critical systems, where the important cost measure is cost per access, while historical data that is not often used should be migrated to high-density SATA disk systems. The concept of flash architectures is that flash will only be a small fraction of the total storage on the system and that it will hold only the fraction of data that is used intensively – for instance active purchase orders data from ERP systems.

This creates a challenge for many businesses, including both large enterprises and SMBs that have never established an effective data tiering system. The ERP databases of many companies contain years of historical data, much of it seldom used but retained for tax, compliance, and decision-support uses. This data slows operations at every turn, creates problems running backups and restores, and takes up space on expensive storage systems. Therefore, before companies can move to flash storage, they need to build efficient tiered data storage architecture first. This means that they need an information lifetime management (ILM) system that can automatically tier data according to the level of demand and/or business rules and archive the older historical data in a way that minimizes space while making it available when needed.

You need a vendor like Solix with the vision to understand the disruptive forces now just starting to impact data storage, including flash, and that is developing new versions of their product sets that will work with the new IT architectures that are evolving today. These are exciting times for data technology. After a decade of essentially quiet slow development, it is being rocked by new physical technologies as well as whole new kinds of database technology such as Hadooop and a tremendous influx of data. Today these sound like exotic technologies, but soon your business’s very survival in a highly competitive marketplace may depend on their successful adoption. It is not too soon to be thinking about them and preparing, and ILM technology will play a vital role. You will reap benefits from increased efficiency, higher service levels, and shorter backup and restore times, immediately.

We had a good year in 2011. We won some significant new customers, signed up new resellers worldwide, made great strides in improving and adding to our product set, and our win-rate continued to increase. We are looking forward to an even better 2012, with upcoming new product announcements and new partnerships. We are excited about the future and wish all our customers and partners a great year ahead.

Here is a case study of one of our customer, how they used Solix EDMS Test Data Management tool set to automate the creation of subsets of production databases for test and development, increasing productivity, saving money, and improving security.

We recently completed an engagement with a large Public Sector customer based in Washington DC, working with an Oracle EBusiness Suite, a 16 TB database. We were able to automate the creation of data subsets for test and development and the resulting database size is 1% (180 GB to be precise) of that database while meeting the requirements of the test environment. This means we saved 99% storage and depending on the number of copies, the storage savings could be significant. If you further consider the savings in backup, and CPU use, along with associated energy and cooling, the overall savings is substantial. And smaller the dataset, faster to load, backup, and restore, and requires fewer hardware resources overall. This also improves security, as this data subset contains lesser data than fully operational database, further sensitive data has been masked by Solix EDMS Data Masking.

An adequate test database must contain data that meets the needs of the selected test cases in sufficient quantity to meet the test requirements. And to test an application correctly, the test environment must match the production system as closely as possible and it must meet the needs of development, testing & training environments, each of which may have a different level of requirements. Determining how much data and what data for each environment, manually is complicated and time consuming, and IT staff lacks the time, many shops simply clone their entire production database for tests and end up creating data breach situations like what Sony is currently facing.

Test Instances for other than for load testing, can work with smaller subsets. In this case, for instance, using a clone of the entire 16 TB database would create tremendous delays in testing. Just loading the full database would take significantly longer than the subset Solix created, and then each test that involved processing the data would be similarly time-consuming. And it would require essentially a full duplicate of the hardware in the production environment, starting with enough storage to hold the 16 TB database. Studies show that 60% of application development and testing time is devoted to data-related tasks that, at best, have only a peripheral relationship to the applications actually being tested. All of this time, and the extra resources required by the full database, are wasted, therefore.

IT organizations looking for ways to improve productivity need to attack this problem. The best answer is to use an automated tool set to capture test data requirements and create a subset of the production database that meets test needs while minimizing the size of the test database. And that toolset should also allow you to mask sensitive data in the dataset, to provide the highest level of security to that data. This is exactly what the Solix EDMS tool set does. Here is a case, where one of the largest data centers with one of the most popular enterprise application is using Solix EDMS Test Data Management.

Clearly the “next big thing” in the data center is virtualization. But we believe that IT shops that virtualize wholesale, without first retiring applications that they no longer need using tools like Solix EDMS or Solix ExAPPS, are missing a large chunk of the potential ROI.

The real benefits of virtualization are the substantial energy, cooling, and space savings, which have allowed organizations to extend the lives of data centers for extra decades. The big offenders in the data center are underutilized servers. An idle server uses 60%-70% of its maximum power and cooling requirements. Thus by raising server utilization from the 15% rate typical in pre-virtualized environments to the 85% rate normally seen in virtualized systems, the CIO can eliminate many of the boxes on the floor, cutting power and cooling demand while making room for future growth.

But virtualizing software that the organization no longer needs is not the best answer for those applications, and actually can make them harder to identify as they get lost in the overall virtualized environment. Often these obsolescent applications are forgotten and only found when someone does a systematic audit of what is running on each server or blade in the data center.

How important is this? Gartner estimates that 10% of the applications in an unoptimized portfolio are candidates for retirement while an additional 33% are candidates for migration or rationalization. Why? All applications eventually outlive their usefulness. Business needs and processes change, new technologies supplant old ones, the enterprise buys new divisions with duplicate processes running on different software. But those old applications are often never shut down. Users still need access to data, some users prefer the old application, others find that it is better for one or two lingering tasks.

Virtualizing these obsolescent applications still leaves them using up valuable resources and costing the enterprise money in software licensing and maintenance, support costs, and associated personnel. The better answer is to use application portfolio management (APM) to identify these candidates for oblivion and retire them first. This can have a major impact on resource use even after virtualization, maximizing ROI. And APM then can become the basis of managing the applications environment in a more sophisticated matter going forward — for instance by identifying applications that might be replaced by SaaS and those that should not be virtualized at all.

The problem, of course, is what happens to the data. And the solution is Solix ExAPPS, which relocates the data from retired applications with its full metadata into its repository with almost 90% compression and makes that data available to authorized users through a single viewer, replacing multiple large applications with a single, efficient tool, taking the ROI for virtualization to the max.