1. Difference between traditional data infrastructure and cloud data infrastructure

   1. Traditional data infrastructure

      1. Hardware such as storage and servers are physically located in organization premises and must be maintained by the organization itself.

      2. Not really scalable as it requires buying and installing new physical hardware.

      3. High upfront costs due to paying for power, space, and IT staff.

   2. Cloud data infrastructure

      1. Hardware is owned and maintained by cloud service provider. Resources can be accessed remotely over the internet.

      2. More scalable as resources can be shut down within minutes. Organization only pay for the capacity they actually use.

      3. No big upfront investment.

2. GCP Service

   1. Data Generation : Cloud Pub/Sub

   2. Data Ingestion : Dataflow

   3. Data Storage : BigQuery

   4. Data Transformation : Dataflow

   5. Data Serving : BigQuery

   
   

3. Key terms

   1. Data warehouse : A centralized repository that stores structured data for analysis, reporting, and business intelligence.

   2. Distributed computing : Method of making multiple computers work together to solve a common problem.

   3. Big data : Refers to data that is large, fast, and complex. Associated with 3Vs (volume, velocity, variety)

   4. Cloud computing : on-demand availability of computing resources and services over the internet.

1. Traditional data infrastructure uses in house servers, storage systems and network devices to collect store, process and manage data. Meanwhile cloud data infrastructure uses cloud storage provided by a third party resource that is more cost effective.

   
   

2. Data generation: Big query

   Data Ingestion: Cloud pub/sub

   Data Storage: Cloud Storage

   Data Transformation: Dataflow

   Data Serving: Cloud SQL

   
   

3. Data Warehouse: centralized system specifically designed for storing and querying large volumes of structured data and typically used for business inteligence

   Distributed computing: is handling compute tasks via a network of computers or servers, rather than relying on a single computer and processor

   Big Data: Big data refers to datasets that are too large, fast, or diverse for traditional data processing tools to handle effectively. It is often defined by 5vs

   Cloud Computing: Cloud computing provides on-demand access to IT services, such as servers, storage, and applications, over the internet, on a pay-per-use basis

Traditional data infrastructure:

Computing relies on local server, hardware, and maintenances. It uses physical data center to store data and assets. Making it hard very costly, limited accessibility.

Cloud data infrastructure:

Services is deliver using internet. A collective combination of configurable systems. Providing more scalable and flexible operation. Ensuring a cost saving IT maintenance.

GCP service for each stages of Data Engineering Lifecycle:

• Data Generation - Raw data (created/extracted)

• Data Ingestion - Cloud Pub/Sub , Cloud Dataflow

• Data Storage - BigQuery, Cloud SQL

• Data Transformation - Dataproc, Cloud Dataflow

• Data Serving - Big Query

Key Terms

• Data warehouse: Helps to aggregates data from various sources into a central data store to optimized for querying and analysis. Enabling organization to centralize data from disparate systems

• Distributed computing: System that consists of multiple software components - on multiple computers but run as a single system.

• Big data: Data volumes exceeding traditional database limits (>1-2TB). It is a massive and complex data that come from diverse sources where traditional data management system cannot handle.

• Cloud Computing: Work as on-demand access to remote server.

• Traditional Data Infrastructure:

  • Runs on on-premise servers and hardware.

  • Requires large upfront investment (CAPEX) in hardware, storage, and maintenance.

  • Scaling is limited and slow since it needs physical resources.

• Cloud Data Infrastructure:

  • Operates on a pay-as-you-go model (OPEX), reducing upfront costs.

  • Highly scalable and flexible, resources can be provisioned instantly.

  • Cloud provider handles maintenance, updates, and much of the security.

2 Five Stages of the Data Engineering Lifecycle with Google Cloud Platform (GCP) Services

• Data Generation – This stage involves the creation of data from various sources such as applications, devices, sensors, or user activities.

  • GCP Service: Cloud Pub/Sub, which provides real-time event streaming and reliable message delivery from diverse data sources.

• Data Ingestion – This stage focuses on collecting and transporting data from its sources into the data pipeline.

  • GCP Service: Cloud Dataflow, which supports both batch and stream processing for Extract, Transform, Load (ETL) tasks.

• Data Storage – At this stage, data is securely stored in suitable repositories for both raw and processed forms.

  • GCP Service: Cloud Storage is used for object and unstructured data, while BigQuery serves as a highly scalable and fully managed data warehouse for structured analytical data.

• Data Transformation – This stage involves cleansing, aggregating and enriching data to make it meaningful and usable for analytics.

  • GCP Service: Cloud Dataflow provides transformation capabilities through stream.

• Data Serving – The final stage ensures that the processed data is accessible for analysis, visualization, and decision-making.

  • GCP Service: BigQuery enables fast, SQL-based analytics on large datasets.

3. Key terms in data engineering evolution

• Data Warehouse: A centralized repository optimized for storing structured data, used for analytics and reporting.

• Distributed Computing: A computing model where tasks are divided across multiple machines to improve processing speed and scalability.

• Big Data: Extremely large and complex datasets (structured, semi-structured, unstructured) that require specialized tools and architectures to process and analyze.

• Cloud Computing: Delivery of computing services (storage, processing, networking, analytics) over the internet, enabling on-demand scalability and cost efficiency

Aidil(IS01082943) :)

1. Traditional data infrastructure is hosted in-house and managed by IT staff, ensuring that security is protected under strict access by the company itself.

   cloud data infrastructure storage managed by a third party or service provider may be exposed to security breaches.

2. Data Generation : BigQuery

   Data Ingestion : Pub/Sub

   Data Storage : Cloud SQL

   Data Transformation : Dataflow

   Data Serving : Cloud Spanner

3. Data Warehouse : A centralized repository that stores structured, historical, and current data from multiple sources.

   
   

   Distributed Computing : A computing model where processing tasks are divided across multiple machines that work together to solve large scale.

   
   

   Big Data : large and diverse datasets that are huge in volume and also rapidly grow in size over time.

   
   

   Cloud Computing : Cloud computing is the delivery of computing services including servers, storage, databases, networking and software over the internet.

Traditional Data Infrastructure

Setup: Physical serves in the building

Scaling: Hard Scaling and takes time

Cost: Big upfront investment

Access: Only from specific locations

Speed of Innovation: Slower and tied to hardware

Cloud Data Infrastructure

Setup: Virtual serves hosted by cloud providers

Scaling: Easy to scale up/down automatically

Cost: Pay for what we are using similar as utility bills

Access: Accessible from anywhere with internet

Speed of Innovation: Faster, with access to new tools and technology

Question 2

Data Generation: Cloud Sub/Pub (Which handles real-time messages)

Data Ingestion: Dataflow (For streaming and batch processing)

Data Storage: Cloud Storage (For raw files)

Big Query (For analysis purposes)

Data Transformation: Dataprep for prepping data

Data Serving: Looker (Dashboards)

Question 3

Data Warehouse

A big, central place to store clean, structured data so teams can run reports and analysis. Example: BigQuery

Distributed Computing

Breaks big tasks into smaller ones and runs them on many computers at once. Example: Apache Spark

Big Data

Data that's too large or messy for regular tools to handle. You need special tools to process and understand it.

Example: high mb/gb videos, pictures and files

Cloud Computing

Using the internet to access computing power, storage, and services – instead of buying and maintaining your own hardware. Example: GCP

Surthikkaa Laavanya (IS01083061)

1. Traditional data infrastructure is owned by organizations, which manage physical servers and data warehouses. Expanding it requires the organization to buy expensive hardware and maintain it through IT teams. Meanwhile, cloud data infrastructure is service-based, where organizations subscribe to providers like AWS or Google Cloud. It offers more flexibility since the cloud vendors handle infrastructure, security, and redundancy.

2. Data Generation: Pub/Sub

   Data Ingestion: Dataflow

   Data Storage: Cloud Storage

   Data Transformation: Dataproc

   Data Serving: BigQuery

3. Data Warehouse: A centralized system for storing structured data from multiple sources, optimized for reporting and analysis

   Distributed Computing: A model where processing is spread across multiple servers/machines to handle very large-scale data

   Big Data: Extremely large and complex datasets that exceed traditional database limits, often characterized by the 3Vs

   Cloud Computing: Delivery of computing resources over the internet which enable scalability and flexibility

(IS01083502)

1. Traditional Data Infra required the organization to manage everything on its own, from hardware, scaling, maintenance, backup, and disaster recovery. Cloud Data Infra is hosted by service providers such as Google, AWS, and Alibaba. It requires no physical hardware to be present cause the host provides everything, the organization can choose the desired resources based on their need, and it can be scaled up and down to fulfill their requirement. It also gives an analytical dashboard and data to the customer

2. Data Generation = Google Cloud pub/sub

   Data ingestion = Cloud Dataflow

   Data Storage = Cloud big table

   Data Transformation = Dataproc

   Data Serving = Looker

3. Data Warehouse = A centralized system to store structured data for analytics and reporting.

   Distributed Computing = Splitting large data tasks across multiple machines working in parallel to ease the workload when processing data.

   Big data = A huge amount of data and a complex dataset that can't be stored by traditional databases. It can be characterized by the 3v

   Cloud Computing = On-demand access to computing, storage, and services over the internet

1. Traditional Data Infrastructure vs Cloud Data Infrastructure :

   
   

   Traditional Data Infrastructure

   i. On-premise (physical servers/data centers)

   ii. Limited and costly to scale

   iii. High upfront costs (hardware, maintenance)

   
   

   Cloud Data Infrastructure

   i. Hosted on cloud platforms (AWS)

   ii. Easily scalable on-demand

   iii. Pay-as-you-go model

   
   

2. Five Stages

   i. Data Generation

   • Description: Creation of raw data from various sources.

   • example : Firebase

   ii. Data Ingestion

   • Description: Bringing data into the system.

   • Example : Cloud Pub/Sub

   iii. Data Storage

   • Description: Storing raw and processed data.

   • Example : Cloud Storage

   iv. Data Transformation

   • Description: Cleaning, enriching, and transforming data.

   • Example : Cloud Dataflow

   v. Data Serving

   • Description: Making data available for analytics, BI, or applications.

   • Example : Looker

     
     

3. Data Warehouse:
   A centralized repository that stores structured data from multiple sources, optimized for fast querying and analytics. Example: BigQuery.

   Distributed Computing:
   A model where computation is spread across multiple machines to handle large workloads efficiently. Frameworks like Apache Hadoop or GCP's Dataflow utilize this.

   Big Data:
   Extremely large datasets that traditional data processing tools can’t handle efficiently. Characteristics are often summarized as the 4 V’s: Volume, Velocity, Variety, and Veracity.

4. Cloud Computing:
   Delivery of computing services (servers, storage, databases, networking, software) over the internet. Enables scalability, flexibility, and cost-efficiency.

   
   

   
   

   
   

   
   

   
   

1.Traditional data infrastructure relies on physical servers and storage owned and maintained by a company, requiring high upfront costs, manual scaling, and ongoing maintenance. Cloud data infrastructure is hosted by providers, offering on-demand resources, easy scalability, lower upfront costs, and pay-as-you-go flexibility without the need to manage hardware.

2.Data Generation – The stage where raw data is created from applications, devices, or users.

• Data Ingestion – The process of collecting and moving data into the system.

• Data Storage – Storing data in a secure and scalable system for later use.

• Data Transformation – Cleaning, structuring, and processing data to make it useful.

• Data Serving – Making processed data available for analysis, reporting, or applications.

3.Data Warehouse – A centralized system that stores structured data from different sources for reporting and analysis.

• Distributed Computing – A method of processing data using multiple computers working together as a single system to handle large tasks efficiently.

• Big Data – Extremely large and complex datasets that cannot be managed or processed by traditional databases, often characterized by high volume, velocity, and variety.

• Cloud Computing – The delivery of computing resources over the internet on a pay-as-you-go basis.

1. Difference between traditional data infrastructure and cloud data infrastructure:

• Traditional data infrastructure is managed on-site, where organizations must buy, install, and maintain their own servers, storage, and networking equipment. This requires higher costs and more effort in maintenance.

• Cloud data infrastructure, however is managed by providers such as Google Cloud. Organizations only need to rent storage and computing resources through the internet. It is more flexible, scalable, and cost-effective since hardware ownership and maintenance are handled by the provider.

2. Five stages of the data engineering lifecycle with GCP services:

• Data Generation: Google Cloud Pub/Sub

• Data Ingestion: Cloud Dataflow

• Data Storage: BigQuery

• Data Transformation: Dataprep or Dataflow

• Data Serving: BigQuery

3. Key terms in data engineering evolution:

   • Data warehouse: A centralized storage system designed for reporting and analysis of structured data

   • Distributed computing: A method where large tasks are divided across multiple computers to work simultaneously, speeding up process

   • Big data: Very large and complex datasets that cannot be processed efficiently with traditional data tools

   • Cloud computing: The delivery of computing services such as storage, servers, and databases over the internet instead of relying on physical hardware

     
     

1. a) One of the differences is how the data is stored. For traditional data infrastructure, it is stored in on-premise relational databases. However, the storage is limited with semi-structured/unstructured data. For data cloud infrastructure, it uses cloud-native storage like Amazon S3 and storage is virtually unlimited and scales automatically.

   
   

   b) Next is how the data is processed. For traditional data infrastructure, ETL jobs run on local servers with tools like Informatica, SSIS, or custom scripts and it has limited parallelism and scalability. For data cloud infrastructure, it uses distributed processing frameworks like Apache Spark and pipelines are often orchestrated with cloud-native tools.

   
   

   c)Lastly, they differ in where they are stored in warehouses. For traditional data infrastructure, it is stored in on-premise data warehouses but scaling requires buying more hardware. For data cloud infrastructure, it uses cloud-native warehouses like Snowflake, BigQuery, Redshift and it has elastic compute & storage scaling

2. 5 stage of data engineering :

   
   

   a) Data generation. Here is where the data is created. GCP can be used is Google Analytics.

   
   

   b) Data ingestion. Here it involves moving data from its source into the cloud. GCP can be used is Cloud Dataflow.

   
   

   c) Data Storage. Here, data is stored for further processing or querying. GCP can be used is Google Cloud Storage.

   
   

   d) Data transformation. Here, it process raw data into clean, usable formats. GCP can be used is Cloud Dataflow

   
   

   e) Data serving. Here, it transform data available to end users, dashboards, ML models, or applications. GCP can be used is Big Query.

3. Key terms definition:

a) Data warehouse: Centralized system used for storing structured data from multiple sources, optimized for querying and reporting.

b) Distributed Computing: Computing model where tasks are divided and run across multiple machines to improve performance, fault tolerance, and scalability.

c) Big Data: Datasets that are too large or complex for traditional systems to store, process, or analyze effectively.

d) Cloud computing: Delivery of computing services over the internet, on a pay-as-you-go basis.

1. traditional data infrastructure requires new hardware, possibly more physical space and cloud data infrastructure has flexible options. Traditional IT systems offer a level of control that some companies prefer. In cloud computing, security measures include encryption, multi-factor authentication, and compliance with various industry standards.

2. i. data generation: Cloud Pub/Sub: Provides low-latency, real-time messaging for event-driven architectures, enabling efficient data generation from diverse sources like IoT devices or application logs.

ii. data storage: Cloud Storage: Offers various storage classes (Standard, Nearline, Coldline, Archive) to optimize cost and performance based on data access patterns and retention needs.

iii. data ingestion: Cloud Dataflow: A fully managed service for executing Apache Beam pipelines, enabling high-throughput, low-latency data ingestion and processing for both batch and stream data.

iv. data transformation: Cloud Dataflow: Processes and transforms data at scale, performing complex ETL (Extract, Transform, Load) operations efficiently.

v. data serving: BigQuery: Serves as a high-performance data source for analytics, reporting, and machine learning models, enabling fast query responses for end-users and applications.

3. i. data warehouse: aggregates data from various sources into a central data store optimized for querying and analysis.

ii. distributed computing: Distributed computing is the method of making multiple computers work together to solve a common problem. It makes a computer network appear as a powerful single computer that provides large-scale resources to deal with complex challenges.

iii. big data: Big data refers to extremely large and complex data sets that cannot be easily managed or analyzed with traditional data processing tools, particularly spreadsheets.

iv: cloud computing: the practice of using a network of remote servers hosted on the internet to store, manage, and process data, rather than a local server or a personal computer.

1. Difference between

Traditional Data Infrastructure:

• Data is stored on physical hardware that's owned and managed by the organisation

• requires high upfront costs & maintenance

• provides maximum control

Cloud Data Infrastructure:

• Data is stored on remote servers provided by third parties

• accesible over the internet

• Cost effective & reduced operational overload

  
  

2. Five stages of Data Engineering Lifecycle & GCP service:

1. Data Generation: Cloud Firestore, Cloud SQL (GCP doesn't directly generate data)

2. Data Ingestion: Cloud Pub/Sub, Cloud Dataflow

3. Data Storage: Cloud Storage, Big Query, Bigtable

4. Data Transformation: Big Query, Dataproc

5. Data Serving: Big Query, Looker

   
   

3. Define:

• Data Warehouse: A central, structured repository that stores & organizes vast amounts of historical data from various sources

• Distributed Computing: Breaks down complex tasks & executed across multiple independent computers that communicate & coordinate with each other over a network. (work together as single system to achieve a common goal)

• Big Data: enormous, complex data sets, characterized by their high Volume, Velocity, and Variety, that traditional software cannot manage.

• Cloud Computing: on-demand, pay-as-you-go use of computing resources like servers, storage & databases over the internet, allowing users to access these services without buying or managing their own physical hardware.

1. Traditional data infrastructure

• Data generated from difference premise apps such as ERP systems, IoT devices and servers.

• Storage are limited by hardware and expensive to scale.

Cloud data infrastructure

• Data generated from cloud-native apps, SaaS, IoT and global digital services.

• Elastic capacity and much more cheaper.

2. 1 GCP for each 5 stages of Data Engineering Lifecycle

1. Data Generation - Google Analytics

2. Data Ingestion - Transfer Service (batch load)

3. Data Storage - Cloud Storage

4. Data Transformation - Dataflow (ETL/ELT pipelines)

5. Data Serving - BigQuery (SQL queries)

3. Key terms in data engineering

1. Data warehouse - Storage place where a company keeps all the the important and organized data.

2. Distributed Computing - Divide the each computer handle a certain part, making the process much faster.

3. Big Data - Large and complex amount of data for normal computer to handle and require specific tools to process it.

4. Cloud Computing - Using the internet to access computing power, storage and software.

1. Traditional data infrastructure relies on on-site physical servers managed at the location of the business itself, which requires expertise to maintain and manage. On the other hand, cloud data infrastructure uses already existing hardware and resources that are managed by third-party cloud providers offering IaaS, PaaS, and SaaS.

2. Data Generation: Dataproc, BigQuery

   Data Ingestion: DataFlow

   Data Storage: Bigtable, Cloud SQL

   Data Transformation: Dataflow

   Data Serving: LookerStudio

3. a) Data warehouse: A data warehouse is a centralised data repository that contains structured data collected from different data resources.

   b) Distributed computing: Distributed computing refers to a model where IT tasks are divided and done by multiple machines that work together as a single system.

   c) Big data: Big data refers to huge and complex datasets that exceed traditional database limits.

   d) Cloud computing: Cloud computing refers to computing resources being offered as a service over the internet.

Question 1: A.Traditional Data Infrastructure

-On-premises setup: Servers and equipment are kept in your own building or office.

-Capital-intensive: investment is required to purchase hardware, licenses, networking equipment, and data center space. Additional ongoing costs include power, cooling, and dedicated IT staff.

-Limited scalability: Scaling up means purchasing and installing more physical servers.

-Manual management: IT teams are responsible for performing updates, applying security patches, managing backups, and monitoring performance often using manual tools

-Rigid architecture: Supports integration across public, private, and hybrid environments. Data and applications can move between systems more easily, supporting modern development workflows like DevOps

-Disaster recovery challenges: Backup and recovery solutions are often complex and costly.

B.Cloud Data Infrastructure

-Hosted remotely: Data and services are stored and managed in cloud providers’ data centers.

-Pay-as-you-go model: Organizations pay only for the resources they use, reducing capital expenditure.

-Highly scalable: Resources can be scaled up or down instantly based on demand.

-Automated management: Cloud platforms offer automated updates, backups, and security features.

-Flexible and integrated: Supports hybrid and multi-cloud environments, enabling seamless data flow.

-Robust disaster recovery: Built-in tools help you recover fast if something goes wrong.

Question 2:

A. Data Generation:

• What happens?
  Data is created by apps, websites, devices (like sensors), or users.

• GCP Tool: Cloud Pub/Sub
  Think of it like a message collector it grabs data as soon as it happens (real-time).

  
  

B. Data Ingestion:

• What happens?
  Data is stored in a way that makes it easy to access later.

• GCP Tools:

  • BigQuery – For structured data (like tables, numbers, etc.)

  • Cloud Storage – For files, images, logs, and other unstructured data.

C. Data Transformation:

• What happens?
  Raw data is cleaned, fixed, and reshaped so it’s ready to analyze.

• GCP Tools:

  • Dataprep (part of Dataplex) – A no-code, drag-and-drop tool for cleaning data

  • Dataflow – Can also be used to transform data with coding (Apache Beam)

D. Data Serving:

• What happens?
  Final, clean data is used in dashboards, reports, or apps.

• GCP Tools:

  • BigQuery – Run fast SQL queries on big data

  • Looker – Build dashboards and visualize insights

Question3:

A. Data Warehouse

• A big, organized place to store data from different sources.

• Example: Google BigQuery – Serverless data warehouse used to analyze large datasets using SQL.

B. Distributed Computing

• Big jobs are split up and shared across many computers.

• Example: Apache Hadoop – Breaks big data into smaller chunks and processes them across clusters.

C. Big Data

• Really large and complex sets of data.

• Comes in all shapes and forms — like text, videos, logs, etc.

• Known for the 3 Vs:

  • Volume – A lot of data

  • Velocity – Data comes in fast

  • Variety – Different types of data

• Needs special tools like Hadoop, Spark, or NoSQL databases.

D. Cloud Computing

• You rent computing power and storage instead of buying it.

• Run apps, store files, and process data online, without owning servers.

• Example: Google Cloud Platform (GCP) – Used for data storage, machine learning, and hosting apps.

Traditional data infrastructure:

-Data is stored physical servers.

-Costly and time consuming

-Less Flexible

Cloud data infrastructure:

-Data is stored on remote servers accessed via the internet

-Data is accessible from anywhere with an internet connection.

1)FireBase IoT Core, Cloud Logging

2)Dataflow

3)BigQuery, Cloud Storage

4)Dataflow

5)Cloud SQL

Data Warehouse: stores structured data from multiple sources for reporting and analysis.

Distributed Computing: processing tasks are divided across multiple machines that work together as a single system.

Big Data: very large data that cannot be easily to manage, analyze and process using traditional methods.

Cloud Computing: servers, storage, databases over the internet.

1)Question 1

Traditional data infrastructure :

• system run on physical servers and storage that a company owns and maintains.

• Scaling them up means buying and installing more hardware, which takes time and money.

• The IT team also has to handle upgrades, maintenance, and security.

Cloud data infrastructure:

• system runs on platforms like Google Cloud, AWS, or Azure.

• It’s flexible, meaning it can scale resources up or down instantly and only pay for what use.

• The cloud provider takes care of most of the maintenance, updates, and security, businesses can focus on using the data instead of managing hardware.

2)Question 2

-Data Generation: Cloud Pub/Sub

-Data Ingestion: Dataflow

-Data Storage: BigQuery

-Data Transformation: Dataflow

-Data Serving: BigQuery,Looker studio

3)Question 3

-Data Warehouse: A single place where data from different sources is collected and structured to making it easier to run reports and analysis.

-Distributed Computing: Breaking big tasks into smaller pieces and running them across many machines at once which makes handling large datasets faster and more efficient.

-Big Data: large and complex datasets that go beyond what traditional databases can handle. It often described using the Volume (size), Variety (different types), and Velocity (speed of data).

-Cloud Computing: Using the internet to access computing power, storage, and services on demand.

1. What is the difference between traditional data infrastructure and cloud data infrastructure?

   • Traditional data infrastructure

     Companies manage their own servers, storage, and databases, giving them full control and direct oversight of their systems.

   • Cloud data infrastructure

     stream-based and runs online through platforms like GCP, offering flexible pay-as-you-go services, automatic scalability, and built-in reliability and security, which makes it easier to support growing data needs and modern analytics.

2. Refer to the five stages of the Data Engineering Lifecycle: Data Generation, Data Ingestion, Data Storage, Data Transformation and Data Serving. For each stage, find at least one Google Cloud Platform (GCP) service that can be used.

   • Data Generation → Firebase,IoT Core.

   • Data Storage → BigQuery,Cloud Storage, Spanner.

   • Data Ingestion → Pub/Sub,Dataflow.

   • Data Transformation → Dataflow,Dataproc ,BigQuery SQL.

   • Data Serving → Looker Studio,Vertex AI.

3. Define key terms in data engineering evolution: data warehouse, distributed computing, big data, cloud computing.

   • Data Warehouse → Centralized system that stores structured and historical data kept separate from operational databases, mainly used for reporting and analytics.

   • Distributed Computing → A way of processing data by splitting tasks across many machines so that big workloads can be done faster and more efficiently.

   • Big Data → Very large, fast, and diverse datasets that traditional databases cannot handle, usually described by the three Vs: volume, velocity, and variety.

   • Cloud Computing → The use of internet-based services that provide computing, storage, and processing power on demand, where you only pay for what you use and can scale easily.

1. Traditional data infrastructure relies on physical servers and storage that are maintained by an organization. Scaling requires purchasing and installing additional hardware, which often results in high upfront costs and longer deployment times.

   Cloud data infrastructure is built on virtualize resources provided by cloud providers such as Google Cloud. It offers on-demand scalability, where organizations can scale resources instantly without investing in physical hardware.

2. i) Data Generation : Cloud Pub/Sub

   ii) Data Ingestion : Cloud Pub/Sub

   iii) Data Storage : BigQuery, Cloud Storage

   iv) Data Transformation : Cloud Dataflow

   v) Data Serving : Looker

3. i) Data warehouse : Centralized system for storing and analyzing structured data from various sources.

   ii) Distributed computing : A method of processing data across multiple machines to handle large-scale workloads.

   iv) Big data : Big and complex data sets, characterized by their Volume, Variety, and Velocity.

   v) Cloud computing : The delivery of computing resources, such as servers, storage, databases, analytics, and machine learning over the internet, rather than relying on local hardware.