SimplicityBI

Menu
  • What we do
      • Digital Transformation Strategy
      • Data Implementation Services
      • Project Services
      • Managed Services
  • Solutions
      • Unified Data
        Platforms
      • Cloud data
        Integration
      • Master Data
        Management (MDM)
      • Data Lake
        DW
      • Data
        Visualization
      • Data
        Analytics
  • Technologies 
      • denodo
        • The Denodo Platform


          All the benefits of data virtualization including the ability to provide real-time access to integrated data across an organization’s diverse data sources, without replicating any data.

          The Denodo Platform offers the broadest access to structured and unstructured data residing in an enterprise, big data, and cloud sources in both batch and real-time, exceeding the performance needs of data-intensive organizations.

          Denodo Cloud Solutions

          Oil and Gas
      • technologies-microsoft-icon
        • Unleash the power in your data

          Reimagine the realm of possibility. Microsoft data platform solutions release the potential hidden in your data - whether it's on-premises, in the cloud, or at the edge - and reveal insights and opportunities to transform your business.Why use the Microsoft data platform

          Fast and Agile
          Work with a flexible data platform that gives you a consistent experience across platforms and gets your innovations to market faster—you can build your apps and then deploy anywhere.

          Built-in Intelligence
          The Microsoft data platform brings AI to your data so you gain deep knowledge about your business and customers like never before. Only Microsoft brings machine learning to database engines and to the edge, for faster predictions and better security.

          Enterprise Proven
          Bring your business to scale while trusting that your security, performance, and availability needs are covered—with an industry-leading total cost of ownership.
      • technologies-ibm-icon-transparent
        • How your business can get smarter



          Analytics

          Gain greater insights and innovate faster


          Cloud
          Control of your cloud should belong to you. SoftLayer can help.


          IT Infrastructure
          Build the foundation for cognitive business



          Services
          Transform your business with our expertise
      • technologies-looker-icon-transparent
        • looker is more than data analytics software, a full platform.

          Bring Data to every part of your business.

          Data Everywhere

          Deliver data directly in the tools your teams use everyday. Bring data into every action and every decision - in Slack, in Salesforce.com, even in your custom applications. Or build new applications on top of the Looker Data Platform to truly customize the experience for your business.

          Analytics evolved

          We believe everybody should have access to reliable analytics to make data-driven decisions. And to deliver on this promise, we had to re-imagine and rebuild how analytics are done from the ground up.
      • snowflake
        • To support today’s data analytics, companies need a data warehouse built for the cloud.


          One that offers rapid deployment, on-demand scalability, and compelling performance at significantly lower cost than existing solutions. Snowflake on Amazon Web Services (AWS) represents a SQL data warehouse built for the cloud.

          Snowflake’s unique architecture natively handles diverse data in a single system, with the elasticity to support any scale of data, workload, and users.
      • striim-platform
        • Striim Enables Modern Cloud Architecture


          Striim is a patented, enterprise-grade platform that offers continuous real-time data ingestion, high-speed in-flight stream processing, and sub-second delivery of data to cloud and on-premises endpoints.

          Striim continuously delivers data where you need it, when you need it, and in the correct format to be immediately available to high-value operational workloads.
      • semarchy
        • Intelligent MDM™


          Semarchy is the Intelligent MDM company. Its xDM platform is an innovation in multi-vector Master Data Management (MDM) that leverages smart algorithms and material design to simplify data stewardship, governance, and integration.

          It is implemented via an agile and iterative approach that delivers business value almost immediately and scales to meet enterprise complexity.
      • google-bigquery-platform
        • Google Bigquery


          A fast, highly scalable, cost-effective, and fully managed cloud data warehouse for analytics, with built-in machine learning.

          BigQuery is Google's serverless, highly scalable, enterprise data warehouse designed to make all your data analysts productive at an unmatched price-performance. Because there is no infrastructure to manage, you can focus on analyzing data to find meaningful insights using familiar SQL without the need for a database administrator.
      • tableau-technology
        • Connect to More Data


          Connect to data on prem or in the cloud—whether it’s big data, a SQL database, a spreadsheet, or cloud apps like Google Analytics and Salesforce.

          Access and combine disparate data without writing code. Power users can pivot, split, and manage metadata to optimize data sources. Analysis begins with data. Get more from yours with Tableau.
  • Insights
      • Events
      • White Papers
      • Blog
      • Webinar
  • Who we are
      • Client Stories
  • Join us
    • Careers
Contact us
Tuesday, 19 June 2018 / Published in Data Virtualization

Data Virtualization and the Logical Data warehouse

Author: Rick F. van der Lans
Date: June 2018

The classic data warehouse architecture, consisting of a chain databases and ETL processes, has served may organizations well the last twenty-five years; see Figure 1. But is it the still the right architecture? For example, is it still the architecture for all our new forms of data usage, such as self-service BI, data science and customer-based apps, and can it easily deal with all forms of big data? For more and more organizations the answer is: No. Many of them have started to look for an alternative, more flexible architecture. The one many have found is the logical data warehouse architecture. This is the topic of this third article in a series on use cases of data virtualization.

Figure 1:

The Logical Data Warehouse Architecture in a Nutshell

The logical data warehouse architecture is an agile architecture for developing BI systems, in which data consumers and data stores are decoupled from each other; see Figure 2. The logical data warehouse architecture presents all the data stored in a heterogeneous set of data stores as a single logical database. In this architecture, data consumers don’t have to be aware of where and how the data is stored. All the details of data storage are hidden for them. They should not have to know or care about whether the data they’re using is coming from a data mart, a data warehouse, or even a production database. They should not have to be aware that data from multiple data stores have to be joined, nor should they know whether they are accessing a SQL database, a Hadoop cluster, a NoSQL database, a web service, or simply one or more flat files. The structure of the data stores is hidden as well; data consumers only see the data in the way that’s convenient for them, and they only see data that is relevant to their task. This is all achieved by decoupling data consumers from data stores.

Figure 2:

Here Comes Data Virtualization

Now, there is not one supernatural tool that can do all the above and magically turn an existing data warehouse architecture into a logical data warehouse one. No silver bullet exists. Several tools are needed to accomplish this, such as a database server, a master data management system, and a data cleansing tool. However, the most important component is the data virtualization server. It’s the driving technology of the entire architecture.

Data virtualization servers support all the right features to develop a logical data warehouse. It provides the right features for data security, scalability, query performance, agile development, reuse of metadata, discovery and search of specifications, big-data access, and so on. But most importantly, it offers a comprehensive abstraction layer that decouples data consumers from data stores.

Features of Data Virtualization Servers

The following features make data virtualization the right technology:

  • On-demand data transformation
  • On-demand data integration
  • On-demand data federation
  • On-demand data cleansing
  • ETL (lite)
  • Data source-aware query optimization
  • Network-aware query optimization
  • Caching
  • Scheduling jobs

Is “Logical” the Right Term?

One important side remark, due to the term logical data warehouse, some have the impression that such an architecture does not require physical data stores at all. They assume that every time when data is queried, the production systems are accessed. This is not the case. For various reasons, data stores are still needed. For example, if a production system doesn’t keep track of historic data, it has to be stored somewhere else, meaning the logical data warehouse architecture needs a separate data store; see also Figure 2. Or, the production system can’t handle the extra workload generated by the data warehouse. In this case, data has to be physically copied to a separate data store. The caching mechanism of the data virtualization server can be used here.

In other words, the word logical in the name logical data warehouse doesn’t mean no physical data stores. It means that we try to minimize the amount of physical data stores. If they are not really needed, they are not developed. The less physical data stores are created and the less data is duplicated, the more flexible the architecture is.

Why the Logical Data Warehouse is Agile?

The technology used to develop classic data warehouse architectures demands that everything is built right the first time. Changing specifications afterwards can be time-consuming and expensive. This is not the case for data virtualization servers. Changing the data structures or the transformation logic of virtual tables only involves changing the specifications. There is no need, for example, to unload and reload tables. Almost all the work that has to be done is simply defining new specifications or changing existing ones. There is no large chain of databases.

Summary

The logical data warehouse architecture is suitable for all our new forms of data usage, such as self-service BI, data science and customer-based apps, and it’s capable of dealing with all forms of big data easily. It is the modern architecture that organizations have been looking for. The heart of this architecture is formed by a data virtualization server, making it a very dominant use case for this technology.

In the fourth article of this series [link to next article], we focus on a less well-known use case of data virtualization namely database migration and acceleration.

For more information on the logical data warehouse, see the following articles and whitepapers:

Developing a Bi-Modal Logical Data Warehouse Architecture Using Data Virtualization
http://www.denodo.com/en/document/whitepaper/developing-bimodal-logical-data-warehouse-architecture-using-data-virtualization

 

Designing a Logical Data Warehouse
http://www.redhat.com/en/resources/designing-logical-data-warehouse-whitepaper

The Logical Data Warehouse Architecture is Tolerant to Change
http://www.linkedin.com/pulse/logical-data-warehouse-architecture-tolerant-changes-van-der-lans

The Logical Data Warehouse Architecture is Not the Same as Data Virtualization
http://www.linkedin.com/pulse/logical-data-warehouse-architecture-same-rick-van-der-lans

  • Tweet

What you can read next

Data Virtualization and Database Migration and Acceleration
Data Virtualization and the Logical Data Lake
Data Virtualization and the 360-Degree Customer View

Leave a Reply Cancel reply

Your email address will not be published.

 

 

“Genius is making complex ideas simple,
not making simple ideas complex”

Albert Einstein

ABOUT SimplicityBI

  • What we do
  • Solutions
  • Technologies 
  • Who we are
  • Insights
  • Join us

GET IN TOUCH

T: +1 (800) 308 8114
Email: contact@simplicitybi.com

SimplicityBI
407 2nd Street SW, Calgary
Alberta, Canada

  • Events
  • White Papers
  • Blog
  • Contact us

© 2022. All rights reserved. Powered by Instalogic Marketing

TOP
×

Get In Touch

Find out how SimplicityBI can impact your bottom line and
elevate your organization’s performance.

  • What we do
    • Back
    • Digital Transformation Strategy
    • Data Implementation Services
    • Project Services
    • Managed Services
    • Back
  • Solutions
    • Back
    • Unified Data Platforms
    • Cloud Data Integration
    • Master Data Managment (MDM)
    • Data Lake – DW
    • Data Virtualization
    • Data Analytics
    • Back
  • Insights
    • Back
    • Events
    • White Papers
    • Blog
    • Webinar
    • Back
  • Technologies 
    • Back
    • Denodo
    • Microsoft Data Platform
    • IBM
    • looker
    • Snowflake
    • Striim
    • Semarchy
    • Google Bigquery
    • Tableau
    • Back
  • Who we are
  • Join us
    • Back
    • Careers
    • Back
  • Contact us