Executive Summary
Companies usually move changed data from source to target by performing extracts
and loads. This approach often involves custom programming, monitoring and moving an entire data volume. The CONNX® DataSync tool solves theses issues with its point-and-click interface and pre-programmed selections. Tasks can be automated during successful and problematic synchronizations using CONNX DataSync’s built-in error monitoring. Companies often need to migrate their data. CONNX DataSync creates a target schema that matches the current schema and transforms the data types
automatically to ANSI/ISO compliant versions, at the same time it transfers the source data to the target. No written code is needed to perform any of the data transformations or data conversions. In summary, CONNX DataSync moves the company’s enterprise data from virtually any source location to virtually any target data source and performs incremental updates from source to target at a fraction of the usual full update time.
Accessing Data From a Variety of Data Sources and Platforms
In the modern-day business environment, large organizations can have a variety of data repositories that operate either in association with off-the-shelf software packages or as custom solutions to business needs. Different departments in an organization might run SAP®
on an Oracle® database, PeopleSoft® on Oracle, or CINCOM SYSTEMS® MRP program on
IBM® VSAM. Some companies run tools that require permanent data stores from multiple
vendors.
Complicating this further, data produced by one department, such as manufacturing, may be
needed by another department, such as finance. The finance department’s analysis tools or
skill sets may require a different database platform than the manufacturing department uses
to store its data.
In addition to data access issues, companies often face the daunting task of migrating a database system to a different platform. Total corporate demand for data may exceed the
bandwidth of the server where the data natively resides.
Approaches to Moving Changed Data
There are four typical approaches to moving changed data between systems:
There are several problems with this approach.
- Moving the entire volume of data can be expensive and time consuming
- The batch process must be monitored for conditions like running out of temporary disk space or losing connections
This approach is inefficient, because it takes time and resources to create each extract re- quest.
This approach sends copies of the source database transaction log to the target database. This log contains a list of actions performed against the database. These transactions are then replayed against the target database. Log shipping can be a cost-effective solution since it only moves changed data and has very little source system impact.
There are several problems with this approach.
Both the source and target database must use the same database systems.
Both the source and target databases must use the same type of transaction log.
A timestamp is a field that marks a record with the date and time of the last action against the record. This approach can be used to move changed data to a new target server.
There are several problems with this approach.
This method doesn’t always work unless the system was designed to access and use the timestamp fields.
All the source tables must include timestamps that are modified each time the table changes.
When an insert, update, or delete occurs, a trigger fires to forward the action to another data store.
There are several problems with this approach.
- This method requires custom programming.
- This method slows down the source database due to trigger activity.
Change Data Capture can address the task of data migration and the issues that extracts and loads pose. This tool captures the source data incremental changes as opposed to full up- dates and delivers them to the target data source (a data warehouse or data mart) which can be used for analysis and production tasks. With Change Data Capture, a snapshot of the source data versus the entire database is delivered to the target, being a data warehouse or data mart.
There are two common methods of Change Data Capture:
Batch-Oriented Change Data Capture takes a snapshot of the source database and duplicates the data into the target database. This is usually done in a batch or on-demand environment
Live Change Data Capture captures changes as they happen in the source data base and performs the exact same changes on the target database. This is
usually done in a live-data or transaction-oriented environment.
 |
|
CONNX DataSync is a batch-oriented (or pull) Change Data Capture tool. It captures incre- mental changes and delvers them at user-defined intervals to a data warehouse or data mart in a fraction of the time required by full updates. It allows users to move a complete snap- shot of pre-defined list of tables into the warehouse. DataSync also incrementally updates the current snapshot of data through a simple-to-use scheduling interface.
Figure 2 illustrates the CONNX DataSync process.
The primary difference between the CONNX DataSync tool and general Change Data Capture is the tool’s unique technique of hash keys to detect which records have been added, deleted, or changed on a scheduled basis. CONNX DataSync uses that information to insert, update, or delete the corresponding data in the data warehouse snapshot tables. Incremental updates are delivered from source to target considerably faster than full updates, because hash keys are stored along with the primary key of each record in a separate CONNX private data store.
The CONNX DataSync tool resolves the extract and load script programming, monitoring,
cost and time issues with point-and-click selections. Built-in error monitoring makes it possible to perform different tasks automatically depending on whether synchronization was or was not completed successfully. Since all data transformations (such as converting from big- endian to little-endian format or from hardware specific time formats to ANSI/ISO SQL for- mats) occur automatically, less manual intervention means reduced time and expense.
With the CONNX DataSync tool, primary keys or all keys associated with a table can be transferred to a reporting server. The same keys can be used on both the target and on the source; you can customize the keys to perform different tasks on the target and on the source.
The CONNX DataSync tool moves only changed data. In a table with millions of rows of thousands of characters each, it is much faster to move a few hundred or a few thousand changes than it is to transfer the entire table. The task groupings can be easily separated into different synchronizations to automatically provide each department with the information they want when they want it.
The CONNX DataSync tool also helps users solve data migration issues.
Migrating a database system between different operating systems and databases usually re- quires a considerable amount of time. With the CONNX DataSync Tool, the data and schema can be easily transferred between systems. The only tasks that need to be performed are selecting the source and target databases and synchronization frequency. Translations used to move the schema and data to the new target are performed automatically.
IT staff often spend hours analyzing and mapping out the required data transformations for a typical data migration. The CONNX DataSync tool can minimize manual analysis and map- ping by creating a target schema to match the source schema, and automatically transforming the source data types to ANSI/ISO-compliant versions, while the data migrates to the tar- get. It performs all of the data transformations and data conversions without adding a single line of code.
CONNX can be used to develop a standards based GUI front end for a company’s old data stores using front ends like Java and VB.NET. After CONNX DataSync is used to create a tar- get database, any standards based tools used to access the source database can be used to access the target database. The standards based tools remain completely unchanged. The only task to be completed is to change the name of the data source.
Extracts Data as a complete snapshot of a predefined list of tables from the source and moves it to the target data warehouse or data mart.
Creates a copy of the source data warehouse and moves this copy into the target data warehouse.
Employs unique technique of hash keys to quickly detect when records are updated, deleted, or inserted. By storing the hash keys along with the primary key of each record in a separate CONNX private data store, the CONNX DataSync tool performs incremental updates from source to target at a fraction of the usual full update time.
Initial full update from source to target may take two to three times longer than the equivalent transfer using FTP (for example: 3.5 hours) because the full update also:
Inserts data into SQL Server Creates CRCs
Stores information on CONNX Store
Transforms the source schema data types to the target schema data types
Other costs are also associated with the initial full update, such as location on a Wide Area Network.
The incremental update usually runs four to ten times faster than the full update, a significant decrease in the synchronization time.
On a scheduled basis, detects which records have been added, deleted, or changed, and inserts, updates, or deletes the corresponding data in the data warehouse snapshot tables.
Each file in the synchronization process may be scheduled independently, or as a group.
On a scheduled basis, purges all prior snapshot data in the data warehouse. A completely new snapshot of the data is placed in the data warehouse. Each file in the synchronization process may be scheduled independently, or as a group.
Automatically performs a complete analysis of all source database schemas and applicable transformations for the target data mart schema.
Automatically creates data type transformations of all data elements from source schema to target schema.
Once the target source is populated, CONNX, the engine powering DataSync, can be used with other tools for ad hoc reporting, application development, and Web development through OLE DB, ODBC, or JDBC drivers. CONNX provides the flexibility of querying either against the data warehouse and data marts, or against the live operational data sources.
Additional copies of a source database can be created for each database system in the enterprise that has been logged as a suitable target database in the CONNX Data Dictionary (CDD). The CONNX DataSync Administrator tool allows you to override the default target database for group synchronization and specify any target database known to the system with two mouse clicks.
CONNX DataSync allows you to automatically detect error conditions and re-sync until a successful sync has been performed.
Additional tasks can be triggered based on synchronization status. A triggered task can be as simple as sending an e-mail to notify successful sync completion, or as complex as calling an ETL (Extract, Transform, Load) tool to perform additional transformations against the synchronized data.
The CONNX DataSync tool offers multi-threaded power to leverage the force of inexpensive modern hardware. Multiple CPU 64-bit machines with thousands of MIPS of throughput can be purchased for a few thousand dollars. Since the CONNX DataSync tool is specifically designed for environments running on multiple machines, this can drastically reduce the load against production systems. If demand warrants, multiple synchronization servers can be added to meet increasing demand. As demand for data expands, you can continue to add inexpensive hardware to meet your needs
In addition to the current database security, CONNX can easily add additional security protection. CONNX not only features table- and column-level security used to restrict access to specific types of information, but also includes row level security. For example, an employee will be allowed access to his/her own personnel record, but not to the records of other employees.
In many cases, the CONNX DataSync tool is up and running a production schedule within an hour of installation. Importing a source database and a target database into the CONNX repository only requires the user to provide the connection information and to select the tables to transfer. A mirror image can then be made of the entire database system or any desired subset. Synchronizations can be run at any given time or on a regular, automated schedule.
Eliminates manual/one-time programming efforts.
Provides option for automatic synchronization between originating and target data sources.
Provides consistent methodology for determining and synchronizing adds, changes and deletes.
Streamlines effort required to populate target data sources Time required for updates is reduced significantly
Eliminates the need to create and maintain “homegrown” solutions. Eliminates the need for extracts.
Lowers the total cost of ownership.
Achieves time savings with incremental updates versus full reloads.
In Figure 4, Change Data Capture Operations within legacy databases are contrasted with full database restores to target relational databases. The results show that incremental updates were between three and twenty times faster than full reloads for these customers.
Whether a company is facing a data migration project or is looking to resolve the issues ex- tracts and loads can pose, CONNX DataSync is the solution. CONNX DataSync moves the
company’s enterprise data from virtually any source location to virtually any target data source and performs incremental updates from source to target in a fraction of the usual full update time.
For more information regarding the CONNX DataSync tool, please call, e-mail or visit the Mobilize Web site:
CONNX Solutions, Inc. 425.609.8458
