The tremendous growth in volumes of data – both traditional structured data and new data types such as Internet-of-Things (IoT) – and the advent of in-memory database technologies like SAP’s HANA and NAND flash storage that are faster but more expensive – has made data archiving mandatory. Companies simply cannot afford to operate as they once did, allowing years of data, much of it seldom used, to accumulate in single-tier databases. The old data clogs systems, hurting performance, and, when that database is running on flash or in-memory, it also becomes prohibitively expensive.
For too long, organizations have debated the best way to deal with the lifecycle of application data. Organizations want to engage in true Information Lifecycle Management while ensuring governance, data security, and operational efficiency. While unstructured data archiving is relatively simple as mostly it is based on date stamp, structured data archiving is complex and multiple methods have been used. Here is what we recommend to our customers:
0-3 years of Application Data – Inline – ILM Partitioning
A great deal of research shows that data that is more than about one year old is essentially inactive, accessed only occasionally for business analysis and compliance audits. Some data, such as telephone call records in telecom industry, becomes inactive even sooner. This data needs to be moved off the tier 0 in-memory or tier 1 NAND flash storage to a lower, less expensive SATA disk tier. The original application, however, still needs access for business analysis, tax reporting, and compliance auditing. The answer is to partition the database, basically creating a separate instance of the data on lower tier media that looks to the application like part of the core database. This trick can improve performance of transactional systems tremendously, allow the business to use its high-cost, solid-state storage for only the most valuable data, and simultaneously support business analysis and auditing of all the data in the original application.
Solix can assist an organization’s IT department with creating partitions that not only can improve application performance and save backup and recovery time but make sense in the larger lifecycle of organizational data. However, this partitioned data continues to reside in the database and does not reduce the load on costly, Tier 1 servers and the infrastructure and utilities needed for them. Partitioning alone is not Information Lifetime Management (ILM). When utilized in the context of business rules created by the organization, it becomes an essential part of the ILM framework. Also, applications such as Oracle E-Business Suite, does not support this method on application modules that are already partitioned by the application.
3-7 years of Application Data – Near line – Database Archiving
For data that is older, three to seven years, Solix Database archiving offers the next step in the ILM framework. Archiving can remove massive amounts of data from the servers, yet the data remains accessible to the business user from the native application. This is done while ensuring data integrity and information governance of the data. Database Archiving helps in improving application performance by reducing the amount of production data that applications must process. This also means faster backups, upgrades, and disaster recovery and reduced cloning time. For many organizations, operational costs, including utilities and the ability to reallocate staff time to more mission-critical operations, will also benefit. The challenge for this method is application upgrades as archive data also needs to be upgraded and kept in sync with the native application.
7+ years of Application Data – Offline – Big Data Archiving
For data older than seven years, Solix Enterprise Archiving allows data to detach from the original application. Built on Apache Hadoop, Solix Enterprise Archiving is highly scalable and utilizes commodity hardware, reducing the need for Tier 1 storage. It is an inexpensive, scale-out storage option for old data that can save on licensing fees and other expenses. This data is still searchable and query-able with user-friendly dashboards, ensuring governance, security, and business needs are met at the highest levels. The main advantage, this creates an Enterprise archive platform and with Enterprise Data Lake, lays the foundation for Next generation enhanced analytics.
Each of the above methods, has implications on:
- Application Upgrade,
- Application Performance,
- Reports Search,
- Backup & DR,
- Operational Efficiency,
- Application Testing, and,
- Meeting Regulatory Compliance Requirements.
Solix is the only vendor today providing support for all the above three methods with our Solix EDMS product. We believe Enterprises should embrace this three tier method for all their application for optimized performance, lower costs and enhanced analytics.