{"id":97,"date":"2022-09-28T19:25:09","date_gmt":"2022-09-28T19:25:09","guid":{"rendered":"https:\/\/sites.psu.edu\/jaredmcuevas\/?p=97"},"modified":"2022-09-28T19:25:09","modified_gmt":"2022-09-28T19:25:09","slug":"topic-3-1-master-data-management","status":"publish","type":"post","link":"https:\/\/jaredmcuevas.com\/?p=97","title":{"rendered":"Topic 3.2 \u2013 Data Warehouses & Data Lakes"},"content":{"rendered":"

Personal Lessons Learned<\/strong>. Primarily working as an end-user and in knowledge management roles within large organizations, learning about the dynamics of modern data management architectures was useful if only to gain initial exposure to the terms and current trends.\u00a0 I plan on revisiting these topics after interviewing some industry experts regarding the practical use of these architectures.<\/p>\n

Introduction<\/strong>. There’s a plethora of information published on data warehouses and data lakes, including two great sources listed below (Amazon and Gartner). Here, we don’t need to restate the definitions, but rather draw on take-aways and highlights contrasts between the two terms -both of which are critical within the modern data management landscape.<\/p>\n

Both data warehouses and data lakes are used to store large amounts of client data. Gartner produced a great article in 2020 identifying similarities, differences, uses, trends, and recommendations for these architectural patterns, cited below.\u00a0 We will highlight some of the key findings here:<\/p>\n

Similarities (Data Warehouses & Data Lakes)<\/strong>:\u00a0 \u00a0Both patterns enable data analysis by providing repositories for large amounts of raw data which can be collected and analyzed.\u00a0 According to Friedman and Heudecker, “both provide an endpoint for collection of transactional, detailed data… specifically to support the execution of analytic workloads<\/em>.”<\/p>\n

Differences (Data Warehouses & Data Lakes):\u00a0 <\/strong>The primary contrast between these two systems is the method and focus.\u00a0 Data warehouses contain “curated” data, or that which has been formatted and standardized to some extent.\u00a0 Data lakes, in contrast, contain data in a generally more raw form which may differ depending on the source.\u00a0 Data lakes are premised on storing information “as-is”.<\/p>\n

I. Data Lakes<\/strong> collect and store “unrefined data…with limited transformation and quality assurance…and events captured from a diverse array of source systems” (Gartner, citation below<\/em>).\u00a0 Enterprises with use cases in “exploratory analysis and data science activities” across multiple types and sources of data (i.e., from mobile phones, internal networks, external sources) will likely find the data lake format to be more useful in enabling their analysis activities.\u00a0 This means creative solutions and previously undefined links and trends – i.e., non-intuitive insights – may be better found using analytics processes with data lakes.<\/p>\n

II.\u00a0 Data warehouses<\/strong> are “database[s] optimized to analyze relational data coming from transactional systems and line of business applications….data is cleaned, enriched, and transformed so it can act as the ‘single source of truth’ that users can trust.” (Amazon, citation below<\/em>)<\/p>\n

Taking Amazon’s definition of data warehousing in mind, there is a close connection between MDM (discussed in blog #3.1) and data warehousing.\u00a0 By standardizing data and eliminating redundancy in entities, MDM (architecture<\/em>) and data warehousing (method<\/em>) can work extremely well together synchronously.<\/p>\n

Key Strategic Findings<\/strong>:<\/p>\n

A.\u00a0 One of the important highlights of these two data architectures is that enterprises are not required to choose one or the other.\u00a0 Because analysis functions are growing increasingly complex and demanding, both architectural systems can be used to provide either (a) disparate or (b) sequential results.\u00a0 Friedman and Heudecker note that “it is equally important to recognize that these architectural patterns can bring more value to the enterprise when used in combination…the data warehouse, data lake and data hub can be combined to work together in an effective architecture.”\u00a0 For example, results from more diverse, less-structured data lake can be transferred and further analyzed in the data warehouse.<\/p>\n

B.\u00a0 It’s important to note that these repositories store data long-term (in contrast to data hubs, which generally pass data on and do not store it).\u00a0 In many cases, stored data may not be current. “Because the data warehouse is used for analytical processing, it contains data reflecting a specific point in time rather than the most current values.” – Amazon.\u00a0 Analysts must account for time-value of data and understand the limitations of real-time data when leveraging data warehouses.<\/p>\n

Sources:
\n(1) “What Is A Data Lake”, Amazon, 2022. https:\/\/aws.amazon.com\/big-data\/datalakes-and-analytics\/what-is-a-data-lake\/
\n(2) “Data Hubs, Data Lakes and Data Warehouses: How They Are Different and Why They Are Better Together”, Gartner, Refreshed 2 June 2021, Published 13 February 2020. https:\/\/www.gartner.com\/document\/3980938?ref=d-linkShare
\n(3) “Ten Steps to Build an Agile Information Architecture”<\/p>\n","protected":false},"excerpt":{"rendered":"

Personal Lessons Learned. Primarily working as an end-user and in knowledge management roles within large organizations, learning about the dynamics of modern data management architectures was useful if only to gain initial exposure to the terms and current trends.\u00a0 I plan on revisiting these topics after interviewing some industry experts regarding the practical use of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-97","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=\/wp\/v2\/posts\/97","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=97"}],"version-history":[{"count":0,"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=\/wp\/v2\/posts\/97\/revisions"}],"wp:attachment":[{"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=97"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=97"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jaredmcuevas.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=97"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}