# data[].attributes.attachment_link.readable_text
Whether you use the internet to learn about a certain topic, complete financial transactions online, order food, etc., data is being generated every single second. The use of social media, online shopping and video streaming services have all added to the increase in the amount of data. A study by Domo estimates that 1.7MB data is created every second for every human being on the planet in 2020. And in order to utilize and get insights from such a huge amount of data - data processing comes into play.

Moving forward, let us understand what is data processing.

What Is Data Processing?

Data in its raw form is not useful to any organization. Data processing is the method of collecting raw data and translating it into usable information. It is usually performed in a step-by-step process by a team of data scientists and data engineers in an organization. The raw data is collected, filtered, sorted, processed, analyzed, stored, and then presented in a readable format.

Data processing is crucial for organizations to create better business strategies and increase their competitive edge. By converting the data into a readable format like graphs, charts, and documents, employees throughout the organization can understand and use the data.

Now that we understood what is data processing, let's understand its cycle process.

Data Processing Cycle

The data processing cycle consists of a series of steps where raw data (input) is fed into a process (CPU) to produce actionable insights (output). Each step is taken in a specific order, but the entire process is repeated in a cyclic manner. The first data processing cycle's output can be stored and fed as the input for the next cycle.

Fig: Data processing cycle ( source)

Generally, there are six main steps in the data processing cycle:

Step 1: Collection

The collection of raw data is the first step of the data processing cycle. The type of raw data collected has a huge impact on the output produced. Hence, raw data should be gathered from defined and accurate sources so that the subsequent findings are valid and usable. Raw data can include monetary figures, website cookies, profit/loss statements of a company, user behavior, etc.

Step 2: Preparation

Data preparation or data cleaning is the process of sorting and filtering the raw data to remove unnecessary and inaccurate data. Raw data is checked for errors, duplication, miscalculations or missing data, and transformed into a suitable form for further analysis and processing. This is done to ensure that only the highest quality data is fed into the processing unit.

Step 3: Input

In this step, the raw data is converted into machine readable form and fed into the processing unit. This can be in the form of data entry through a keyboard, scanner or any other input source.

Step 4: Data Processing

In this step, the raw data is subjected to various data processing methods using machine learning and artificial intelligence algorithms to generate a desirable output. This step may vary slightly from process to process depending on the source of data being processed (data lakes, online databases, connected devices, etc.) and the intended use of the output.

Step 5: Output

The data is finally transmitted and displayed to the user in a readable form like graphs, tables, vector files, audio, video, documents, etc. This output can be stored and further processed in the next data processing cycle.

Step 6: Storage

The last step of the data processing cycle is storage, where data and metadata are stored for further use. This allows for quick access and retrieval of information whenever needed, and also allows it to be used as input in the next data processing cycle directly.

Now that we have learned what is data processing and its cycle, now we can look at the types.

Types of Data Processing

There are different types of data processing based on the source of data and the steps taken by the processing unit to generate an output. There is no one-size-fits-all method that can be used for processing raw data.

Type

Uses

Time-sharing

Allocates computer resources and data in time slots to several users simultaneously.

Data Processing Methods

There are three main data processing methods - manual, mechanical and electronic.

Manual Data Processing

In this data processing method, data is processed manually. The entire process of data collection, filtering, sorting, calculation and other logical operations are all done with human intervention without the use of any other electronic device or automation software. It is a low-cost method and requires little to no tools, but produces high errors, high labor costs and lots of time.

Mechanical Data Processing

Data is processed mechanically through the use of devices and machines. These can include simple devices such as calculators, typewriters, printing press, etc. Simple data processing operations can be achieved with this method. It has much lesser errors than manual data processing, but the increase of data has made this method more complex and difficult.

Electronic Data Processing

Data is processed with modern technologies using data processing software and programs. A set of instructions is given to the software to process the data and yield output. This method is the most expensive but provides the fastest processing speeds with the highest reliability and accuracy of output.

Examples of Data Processing

Data processing occurs in our daily lives whether we may be aware of it or not. Here are some real-life examples of data processing:

A stock trading software that converts millions of stock data into a simple graph

An e-commerce company uses the search history of customers to recommend similar products

A digital marketing company uses demographic data of people to strategize location-specific campaigns

A self-driving car uses real-time data from sensors to detect if there are pedestrians and other cars on the road

That was all about the article what is data processing.

Want to begin your career as a Big Data Engineer? Check out the Big Data Engineer Training Course and get certified.

Here's What You Can Do Next

Data contains a lot of useful information for organizations, researchers, institutions, and individual users. With the increase in the amount of data being generated every day, there is a need for more data scientists and data engineers to help understand these data. Simplilearn's Data Engineering Certification Course in collaboration with IBM and partnership with Purdue University offers the highest learning experience to help you master crucial data engineering skills. By leveraging Purdue University's academic excellence in data engineering and IBM's industry-relevant and hands-on training experience, this program will help fast-track your career as a data engineering professional.

Hope you liked the article "what is data processing" in case of doubts ask your questions in the comment section.

What Is Data Processing?

Data in its raw form is not useful to any organization. Data processing is the method of collecting raw data and translating it into usable information. It is usually performed in a step-by-step process by a team of data scientists and data engineers in an organization. The raw data is collected, filtered, sorted, processed, analyzed, stored, and then presented in a readable format.

Data processing is crucial for organizations to create better business strategies and increase their competitive edge. By converting the data into a readable format like graphs, charts, and documents, employees throughout the organization can understand and use the data.

Post Graduate Program in Data Engineering

Your Gateway To Becoming a Data Engineering Expert View Course

Now that we understood what is data processing, let's understand its cycle process.

Data Processing Cycle

The data processing cycle consists of a series of steps where raw data (input) is fed into a process (CPU) to produce actionable insights (output). Each step is taken in a specific order, but the entire process is repeated in a cyclic manner. The first data processing cycle's output can be stored and fed as the input for the next cycle.

Fig: Data processing cycle ( source)

Generally, there are six main steps in the data processing cycle:

Step 1: Collection

The collection of raw data is the first step of the data processing cycle. The type of raw data collected has a huge impact on the output produced. Hence, raw data should be gathered from defined and accurate sources so that the subsequent findings are valid and usable. Raw data can include monetary figures, website cookies, profit/loss statements of a company, user behavior, etc.

Step 2: Preparation

Data preparation or data cleaning is the process of sorting and filtering the raw data to remove unnecessary and inaccurate data. Raw data is checked for errors, duplication, miscalculations or missing data, and transformed into a suitable form for further analysis and processing. This is done to ensure that only the highest quality data is fed into the processing unit.

Step 3: Input

In this step, the raw data is converted into machine readable form and fed into the processing unit. This can be in the form of data entry through a keyboard, scanner or any other input source.

Step 4: Data Processing

In this step, the raw data is subjected to various data processing methods using machine learning and artificial intelligence algorithms to generate a desirable output. This step may vary slightly from process to process depending on the source of data being processed (data lakes, online databases, connected devices, etc.) and the intended use of the output.

Step 5: Output

The data is finally transmitted and displayed to the user in a readable form like graphs, tables, vector files, audio, video, documents, etc. This output can be stored and further processed in the next data processing cycle.

Step 6: Storage

The last step of the data processing cycle is storage, where data and metadata are stored for further use. This allows for quick access and retrieval of information whenever needed, and also allows it to be used as input in the next data processing cycle directly.

Free Course: Big Data Hadoop and Spark Developer

Learn Big Data Basics from Top Experts - for FREE Enrol Now

Now that we have learned what is data processing and its cycle, now we can look at the types.

Types of Data Processing

There are different types of data processing based on the source of data and the steps taken by the processing unit to generate an output. There is no one-size-fits-all method that can be used for processing raw data.

Type

Uses

Time-sharing

Allocates computer resources and data in time slots to several users simultaneously.

Data Processing Methods

There are three main data processing methods - manual, mechanical and electronic.

Manual Data Processing

In this data processing method, data is processed manually. The entire process of data collection, filtering, sorting, calculation and other logical operations are all done with human intervention without the use of any other electronic device or automation software. It is a low-cost method and requires little to no tools, but produces high errors, high labor costs and lots of time.

Mechanical Data Processing

Data is processed mechanically through the use of devices and machines. These can include simple devices such as calculators, typewriters, printing press, etc. Simple data processing operations can be achieved with this method. It has much lesser errors than manual data processing, but the increase of data has made this method more complex and difficult.

Learn Data Science with R for FREE

Master Basics of Data Science with R for FREE Enrol Now

Electronic Data Processing

Data is processed with modern technologies using data processing software and programs. A set of instructions is given to the software to process the data and yield output. This method is the most expensive but provides the fastest processing speeds with the highest reliability and accuracy of output.

Examples of Data Processing

Data processing occurs in our daily lives whether we may be aware of it or not. Here are some real-life examples of data processing:

A stock trading software that converts millions of stock data into a simple graph

An e-commerce company uses the search history of customers to recommend similar products

A digital marketing company uses demographic data of people to strategize location-specific campaigns

A self-driving car uses real-time data from sensors to detect if there are pedestrians and other cars on the road

That was all about the article what is data processing.

Want to begin your career as a Big Data Engineer? Check out the Big Data Engineer Training Course and get certified.

Here's What You Can Do Next

Data contains a lot of useful information for organizations, researchers, institutions, and individual users. With the increase in the amount of data being generated every day, there is a need for more data scientists and data engineers to help understand these data. Simplilearn's Data Engineering Certification Course in collaboration with IBM and partnership with Purdue University offers the highest learning experience to help you master crucial data engineering skills. By leveraging Purdue University's academic excellence in data engineering and IBM's industry-relevant and hands-on training experience, this program will help fast-track your career as a data engineering professional.

Hope you liked the article "what is data processing" in case of doubts ask your questions in the comment section.

# data[].attributes.attachment_link.title
What Is Data Processing: Definition, Cycle, Types & Methods | Simplilearn

# data[].attributes.attachment_link.description
Whether you use the internet to learn about a certain topic, complete financial transactions online, order food, etc., data is being generated every single second. The use of social media, online shopping and video streaming services have all added to the increase in the amount of data.

# data[].attributes.body
how is data stored?

# data[].attributes.body
a collection of interrelated data's stored in a database server

# data[].attributes.body
each file is independent of other file and data in the different file can be integrated only by writing an individual program for each application

# data[].attributes.attachment_link.title
Báo cáo thuyết trình tiếng anh chuyên ngành

# data[].attributes.attachment_link.title
Báo cáo thuyết trình tiếng anh chuyên ngành

# data[].attributes.attachment_link.title
Data storage: Everything you need to know about emerging technologies

# data[].attributes.attachment_link.description
The era of data-centric computing is here -- and, fortunately, data storage is more cost effective than ever. But we’ll see more change in the next decade than we’ve ever seen before in computer data storage. Here’s what’s coming that you need to know.

# data[].attributes.attachment_link.readable_text
Data management is the process of ingesting, storing, organizing and maintaining the data created and collected by an organization. Effective data management is a crucial piece of deploying the IT systems that run business applications and provide analytical information to help drive operational decision-making and strategic planning by corporate executives, business managers and other end users.

The data management process includes a combination of different functions that collectively aim to make sure the data in corporate systems is accurate, available and accessible. Most of the required work is done by IT and data management teams, but business users typically also participate in some parts of the process to ensure that the data meets their needs and to get them on board with policies governing its use.

This comprehensive guide to data management further explains what it is and provides insight on the individual disciplines it includes, best practices for managing data, challenges that organizations face and the business benefits of a successful data management strategy. You'll also find an overview of data management tools and techniques. Click through the hyperlinks on the page to read more articles about data management trends and get expert advice on managing corporate data.

Importance of data management

Data increasingly is seen as a corporate asset that can be used to make better-informed business decisions, improve marketing campaigns, optimize business operations and reduce costs, all with the goal of increasing revenue and profits. But a lack of proper data management can saddle organizations with incompatible data silos, inconsistent data sets and data quality problems that limit their ability to run business intelligence (BI) and analytics applications -- or, worse, lead to faulty findings.

Data management has also grown in importance as businesses are subjected to an increasing number of regulatory compliance requirements, including data privacy and protection laws such as GDPR and the California Consumer Privacy Act (CCPA). In addition, companies are capturing ever-larger volumes of data and a wider variety of data types -- both hallmarks of the big data systems many have deployed. Without good data management, such environments can become unwieldy and hard to navigate.

Types of data management functions

The separate disciplines that are part of the overall data management process cover a series of steps, from data processing and storage to governance of how data is formatted and used in operational and analytical systems. Developing a data architecture is often the first step, particularly in large organizations with lots of data to manage. A data architecture provides a blueprint for managing data and deploying databases and other data platforms, including specific technologies to fit individual applications.

Databases are the most common platform used to hold corporate data. They contain a collection of data that's organized so it can be accessed, updated and managed. They're used in both transaction processing systems that create operational data, such as customer records and sales orders, and data warehouses, which store consolidated data sets from business systems for BI and analytics.

That makes database administration a core data management function. Once databases have been set up, performance monitoring and tuning must be done to maintain acceptable response times on database queries that users run to get information from the data stored in them. Other administrative tasks include database design, configuration, installation and updates; data security; database backup and recovery; and application of software upgrades and security patches.

Data management involves a variety of interrelated functions.

The primary technology used to deploy and administer databases is a database management system ( DBMS), which is software that acts as an interface between the databases it controls and the database administrators (DBAs), end users and applications that access them. Alternative data platforms to databases include file systems and cloud object storage services, which store data in less structured ways than mainstream databases do, offering more flexibility on the types of data that can be stored and how the data is formatted. As a result, though, they aren't a good fit for transactional applications.

Other fundamental data management disciplines include the following:

data modeling, which diagrams the relationships between data elements and how data flows through systems;

data integration, which combines data from different data sources for operational and analytical uses;

data governance, which sets policies and procedures to ensure data is consistent throughout an organization;

data quality management, which aims to fix data errors and inconsistencies; and

master data management (MDM), which creates a common set of reference data on things like customers and products.

Data management tools and techniques

A wide range of technologies, tools and techniques can be employed as part of the data management process. The following options are available for different aspects of managing data.

Database management systems. The most prevalent type of DBMS is the relational database management system. Relational databases organize data into tables with rows and columns that contain database records. Related records in different tables can be connected through the use of primary and foreign keys, avoiding the need to create duplicate data entries. Relational databases are built around the SQL programming language and a rigid data model best suited to structured transaction data. That and their support for the ACID transaction properties -- atomicity, consistency, isolation and durability -- have made them the top database choice for transaction processing applications.

However, other types of DBMS technologies have emerged as viable options for different kinds of data workloads. Most are categorized as NoSQL databases, which don't impose rigid requirements on data models and database schemas. As a result, they can store unstructured and semistructured data, such as sensor data, internet clickstream records and network, server and application logs.

There are four main types of NoSQL systems:

document databases that store data elements in document-like structures;

key-value databases that pair unique keys and associated values;

wide-column stores with tables that have a large number of columns; and

graph databases that connect related data elements in a graph format.

The NoSQL name has become something of a misnomer, though -- while NoSQL databases don't rely on SQL, many now support elements of it and offer some level of ACID compliance.

Additional database and DBMS options include in-memory databases that store data in a server's memory instead of on disk to accelerate I/O performance and columnar databases that are geared to analytics applications. Hierarchical databases that run on mainframes and predate the development of relational and NoSQL systems are also still available for use. Users can deploy databases in on-premises or cloud-based systems. In addition, various database vendors offer managed cloud database services, in which they handle database deployment, configuration and administration for users.

Big data management. NoSQL databases are often used in big data deployments because of their ability to store and manage various data types. Big data environments are also commonly built around open source technologies such as Hadoop, a distributed processing framework with a file system that runs across clusters of commodity servers; its associated HBase database; the Spark processing engine; and the Kafka, Flink and Storm stream processing platforms. Increasingly, big data systems are being deployed in the cloud, using object storage such as Amazon Simple Storage Service (S3).

Data warehouses and data lakes. The two most widely used repositories for managing analytics data are data warehouses and data lakes. A data warehouse -- the more traditional method -- typically is based on a relational or columnar database, and it stores structured data that has been pulled together from different operational systems and prepared for analysis. The primary data warehouse use cases are BI querying and enterprise reporting, which enable business analysts and executives to analyze sales, inventory management and other KPIs.

An enterprise data warehouse includes data from business systems across an organization. In large companies, individual subsidiaries and business units with management autonomy may build their own data warehouses. Data marts are another warehousing option -- they're smaller versions of data warehouses that contain subsets of an organization's data for specific departments or groups of users. In one deployment approach, an existing data warehouse is used to create different data marts; in another, the data marts are built first and then used to populate a data warehouse.

Data lakes, on the other hand, store pools of big data for use in predictive modeling, machine learning and other advanced analytics applications. Initially, they were most commonly built on Hadoop clusters, but S3 and other cloud object storage services are increasingly being used for data lake deployments. They're sometimes also deployed on NoSQL databases, and different platforms can be combined in a distributed data lake environment. The data may be processed for analysis when it's ingested, but a data lake often contains raw data stored as is. In that case, data scientists and other analysts typically do their own data preparation work for specific analytical uses.

A third platform option for storing and processing analytical data has also emerged: the data lakehouse. As its name indicates, it combines elements of data lakes and data warehouses, merging the flexible data storage, scalability and lower cost of a data lake with the querying capabilities and more rigorous data management structure of a data warehouse.

Data integration. The most widely used data integration technique is extract, transform and load (ETL), which pulls data from source systems, converts it into a consistent format and then loads the integrated data into a data warehouse or other target system. However, data integration platforms now also support a variety of other integration methods. That includes extract, load and transform (ELT), a variation on ETL that leaves data in its original form when it's loaded into the target platform. ELT is a common choice for data integration in data lakes and other big data systems.

ETL and ELT are batch integration processes that run at scheduled intervals. Data management teams can also do real-time data integration, using methods such as change data capture, which applies changes to the data in databases to a data warehouse or other repository, and streaming data integration, which integrates streams of real-time data on a continuous basis. Data virtualization is another integration option that uses an abstraction layer to create a virtual view of data from different systems for end users instead of physically loading the data into a data warehouse.

This shows the different approaches that can be used to integrate data.

Data modeling. Data modelers create a series of conceptual, logical and physical data models that document data sets and workflows in a visual form and map them to business requirements for transaction processing and analytics. Common techniques for modeling data include the development of entity relationship diagrams, data mappings and schemas in a variety of model types. Data models often must be updated when new data sources are added or when an organization's information needs change.

Data governance, data quality and MDM. Data governance is primarily an organizational process; software products that can help manage data governance programs are available, but they're an optional element. While governance programs may be managed by data management professionals, they usually include a data governance council made up of business executives who collectively make decisions on common data definitions and corporate standards for creating, formatting and using data.

Another key aspect of governance initiatives is data stewardship, which involves overseeing data sets and ensuring that end users comply with the approved data policies. Data steward can be either a full- or part-time position, depending on the size of an organization and the scope of its governance program. Data stewards can also come from both business operations and the IT department; either way, a close knowledge of the data they oversee is normally a prerequisite.

Data governance is closely associated with data quality improvement efforts. Ensuring that data quality levels are high is a key part of effective data governance, and metrics that document improvements in the quality of an organization's data are central to demonstrating the business value of governance programs. Key data quality techniques supported by various software tools include the following:

data profiling, which scans data sets to identify outlier values that might be errors;

data cleansing, also known as data scrubbing, which fixes data errors by modifying or deleting bad data; and

data validation, which checks data against preset quality rules.

Master data management is also affiliated with data governance and data quality management, although MDM hasn't been adopted as widely as they have. That's partly due to the complexity of MDM programs, which mostly limits them to large organizations. MDM creates a central registry of master data for selected data domains -- what's often called a golden record. The master data is stored in an MDM hub, which feeds the data to analytical systems for consistent enterprise reporting and analysis. If desired, the hub can also push updated master data back to source systems.

Data observability is an emerging process that can augment data quality and data governance initiatives by providing a more complete picture of data health in an organization. Adapted from observability practices in IT systems, data observability monitors data pipelines and data sets, identifying issues that need to be addressed. Data observability tools can be used to automate monitoring, alerting and root cause analysis procedures and to plan and prioritize problem-resolution work.

Data management best practices

These are some best practices to help keep the data management process on the right track in an organization.

Make data governance and data quality top priorities. A strong data governance program is a critical component of effective data management strategies, especially in organizations with distributed data environments that include a diverse set of systems. A strong focus on data quality is also a must. In both cases, though, IT and data management teams can't go it alone. Business executives and users must be involved to make sure their data needs are met and data quality problems aren't perpetuated. The same applies to data modeling projects.

Be smart about deploying data management platforms. The multitude of databases and other data platforms that are available to use requires a careful approach when designing an architecture and evaluating and selecting technologies. IT and data managers must be sure the data management systems they implement are fit for the intended purpose and will deliver the data processing capabilities and analytics information required by an organization's business operations.

Be sure you can meet business and user needs, now and in the future. Data environments aren't static -- new data sources are added, existing data sets change and business needs for data evolve. To keep up, data management must be able to adapt to changing requirements. For example, data teams need to work closely with end users in building and updating data pipelines to ensure that they include all of the required data on an ongoing basis. A DataOps process might help -- it's a collaborative approach to developing data systems and pipelines that's derived from a combination of DevOps, Agile software development and lean manufacturing methodologies. DataOps brings together data managers and users to automate workflows, improve communication and accelerate data delivery.

DAMA International, the Data Governance Professionals Organization and other industry groups also offer best-practices guidance and educational resources on data management disciplines. For example, DAMA has published DAMA-DMBOK: Data Management Body of Knowledge, a reference book that attempts to define a standard view of data management functions and methods. Commonly referred to as the DMBOK, it was first published in 2009, and a second edition, the DMBOK2, was released in 2017.

Data management risks and challenges

Ever-increasing data volumes complicate the data management process, especially when a mix of structured, semistructured and unstructured data is involved. Also, if an organization doesn't have a well-designed data architecture, it can end up with siloed systems that are difficult to integrate and manage in a coordinated way. That makes it harder to ensure that data sets are accurate and consistent across all data platforms.

Even in better-planned environments, enabling data scientists and other analysts to find and access relevant data can be a challenge, especially when the data is spread across various databases and big data systems. To help make data more accessible, many data management teams are creating data catalogs that document what's available in systems and typically include business glossaries, metadata-driven data dictionaries and data lineage records.

The accelerating shift to the cloud can ease some aspects of data management work, but it also creates new challenges. For example, migrating to cloud databases can be complicated for organizations that need to move data and processing workloads from existing on-premises systems. Costs are another big issue in the cloud: The use of cloud systems and managed services must be monitored closely to make sure data processing bills don't exceed the budgeted amounts.

Many data management teams are now among the employees who are accountable for protecting corporate data security and limiting potential legal liabilities for data breaches or misuse of data. Data managers need to help ensure compliance with both government and industry regulations on data security, privacy and usage.

That has become a more pressing concern with the passage of GDPR, the European Union's data privacy law that took effect in May 2018, and the CCPA, which was signed into law in 2018 and became effective at the start of 2020. The provisions of CCPA were later expanded by the California Privacy Rights Act, a ballot measure that was approved by the state's voters in November 2020 and took effect on Jan. 1, 2023.

Data management tasks and roles

The data management process involves a wide range of tasks, duties and skills. In smaller organizations with limited resources, individual workers may handle multiple roles. But in larger ones, data management teams commonly include data architects, data modelers, DBAs, database developers, data administrators, data quality analysts and engineers, and ETL developers. Another role that's being seen more often is the data warehouse analyst, who helps manage the data in a data warehouse and builds analytical data models for business users.

Here are some basic details about the data management profession.

Data scientists, other data analysts and data engineers, who help build data pipelines and prepare data for analysis, might also be part of a data management team. In other cases, they're on a separate data science or analytics team. Even then, though, they typically handle some data management tasks themselves, especially in data lakes with raw data that needs to be filtered and prepared for specific analytics uses.

Likewise, application developers sometimes help deploy and manage big data platforms, which require new skills overall compared to relational database systems. As a result, organizations might need to hire new workers or retrain traditional DBAs to meet their big data management needs.

Data governance managers and data stewards qualify as data management professionals, too. But they're usually part of a separate data governance team.

Benefits of good data management

A well-executed data management strategy can benefit organizations in various ways:

It can help companies gain potential competitive advantages over their business rivals, both by improving operational effectiveness and enabling better decision-making.

Organizations with well-managed data can become more agile, making it possible to spot market trends and move to take advantage of new business opportunities more quickly.

Effective data management can also help companies avoid data breaches, data collection missteps and other data security and privacy issues that could damage their reputation, add unexpected costs and put them in legal jeopardy.

Ultimately, a solid approach to data management can provide better business performance by helping to improve business strategies and processes.

Data management history, evolution and trends

The first flowering of data management was largely driven by IT professionals who focused on solving the problem of garbage in, garbage out in the earliest computers after recognizing that the machines reached false conclusions because they were fed inaccurate or inadequate data. Mainframe-based hierarchical databases became available in the 1960s, bringing more formality to the burgeoning process of managing data.

The relational database emerged in the 1970s and cemented its place at the center of the data management ecosystem during the 1980s. The idea of the data warehouse was conceived late in that decade, and early adopters of the concept began deploying data warehouses in the mid-1990s. By the early 2000s, relational software was a dominant technology, with a virtual lock on database deployments.

But the initial release of Hadoop became available in 2006 and was followed by the Spark processing engine and various other big data technologies. A range of NoSQL databases also started to become available in the same time frame. While relational platforms are still the most widely used data store by far, the rise of big data and NoSQL alternatives and the data lake environments they enable have given organizations a broader set of data management choices. The addition of the data lakehouse concept in 2017 further expanded the options.

But all of those choices have made many data environments more complex. That's spurring the development of new technologies and processes designed to help make them easier to manage. In addition to data observability, they include data fabric, an architectural framework that aims to better unify data assets by automating integration processes and making them reusable, and data mesh, a decentralized architecture that gives data ownership and management responsibilities to individual business domains, with federated governance to agree on organizational standards and policies.

None of those three approaches is widely used yet, though. In its 2022 Hype Cycle report on new data management technologies, consulting firm Gartner said each has been adopted by less than 5% of its target user audience. Gartner predicted that data fabric and data observability are both five to 10 years away from reaching full maturity and mainstream adoption, but it said they could ultimately be very beneficial to users. It was less bullish about data mesh, giving that a "Low" potential benefit rating.

The following are some other notable data management trends:

Cloud data management technologies are becoming pervasive. Gartner has forecasted that cloud databases will account for 50% of overall DBMS revenue in 2022. In the Hype Cycle report, it said organizations are also "moving rapidly" to deploy emerging data management technologies in the cloud. For companies that aren't ready to fully migrate, hybrid cloud architectures that combine cloud and on-premises systems -- for example, hybrid data warehouse environments -- are also an option.

Augmented data management capabilities also aim to help streamline processes. Software vendors are adding augmented functionality for data quality, database management, data integration and data cataloging that uses AI and machine learning technologies to automate repetitive tasks, identify issues and suggest actions.

The growth of edge computing is creating new data management needs. As organizations increasingly use remote sensors and IoT devices to collect and process data as part of edge computing environments, some vendors are also developing edge data management capabilities for endpoint devices.

Continue Reading About What is data management and why is it important?

Dig Deeper on Data management strategies

# data[].attributes.attachment_link.title
What Is Data Management and Why Is It Important?

# data[].attributes.attachment_link.description
Data management is the process of ingesting, storing, organizing and maintaining the data created and collected by an organization. Effective data management is a crucial piece of deploying the IT systems that run business applications and provide analytical information to help drive operational decision-making and strategic planning by corporate executives, business managers and other end users.

# data[].attributes.body
cái này thì chi tiết hơn về các hệ cơ sở dữ liệu nhé

# data[].attributes.attachment_link.readable_text
Database is any collection of electronic records that can be processed to produce useful information. The data can be accessed, modified, managed, controlled and organized to perform various data-processing operations. The data is typically indexed across rows, columns and tables that make workload processing and data querying efficient. Different types of databases include:

In enterprise applications, databases involve mission-critical, security-sensitive, and compliance-focused record items that have complicated logical relationships with other datasets and grow exponentially over time as the user base increases. As a result, these organizations require technology solutions to maintain, secure, manage, and process the data stored in databases.

This is where Database Management Systems come into play.

What is DBMS?

Database Management Systems (DBMS) refer to the technology solution used to optimize and manage the storage and retrieval of data from databases. DBMS offers a systematic approach to manage databases via an interface for users as well as workloads accessing the databases via apps.

The management responsibilities for DBMS encompass:

Information within the databases

The processes applied to databases, such as access and modification

The database's logic structure

DBMS also facilitates additional administrative operations such as change management, disaster recovery, compliance, and performance monitoring, among others.

Components of DBMS

In order to facilitate these functions, DBMS has the following key components:

Software. DBMS is primarily a software system that can be considered as a management console or an interface to interact with and manage databases. The interfacing also spreads across real-world physical systems that contribute data to the backend databases. The OS, networking software, and the hardware infrastructure is involved in creating, accessing, managing, and processing the databases.

Data. DBMS contains operational data, access to database records and metadata as a resource to perform the necessary functionality. The data may include files with such as index files, administrative information, and data dictionaries used to represent data flows, ownership, structure, and relationships to other records or objects.

Procedures. While not a part of the DBMS software, procedures can be considered as instructions on using DBMS. The documented guidelines assist users in designing, modifying, managing, and processing databases.

Database languages. These are components of the DBMS used to access, modify, store, and retrieve data items from databases; specify database schema; control user access; and perform other associated database management operations. Types of DBMS languages include Data Definition Language (DDL), Data Manipulation Language (DML), Database Access Language (DAL) and Data Control Language (DCL).

Query processor. As a fundamental component of the DBMS, the query processor acts as an intermediary between users and the DBMS data engine in order to communicate query requests. When users enter an instruction in SQL language, the command is executed from the high-level language instruction to a low-level language that the underlying machine can understand and process to perform the appropriate DBMS functionality. In addition to instruction parsing and translation, the query processor also optimizes queries to ensure fast processing and accurate results.

Runtime database manager. A centralized management component of DBMS that handles functionality associated with runtime data, which is commonly used for context-based database access. This component checks for user authorization to request the query; processes the approved queries; devises an optimal strategy for query execution; supports concurrency so that multiple users can simultaneously work on same databases; and ensures integrity of data recorded into the databases.

Database manager. Unlike the runtime database manager that handles queries and data at runtime, the database manager performs DBMS functionality associated with the data within databases. Database manager allows a set of commands to perform different DBMS operations that include creating, deleting, backup, restoring, cloning, and other database maintenance tasks. The database manager may also be used to update the database with patches from vendors.

Database engine. This is the core software component within the DBMS solution that performs the core functions associated with data storage and retrieval. A database engine is also accessible via APIs that allow users or apps to create, read, write, and delete records in databases.

Reporting. The report generator extracts useful information from DBMS files and displays it in structured format based on defined specifications. This information may be used for further analysis, decision making, or business intelligence.

DBMS system schematic

The following diagram illustrates the schematic of a DBMS system:

Benefits of DBMS

DBMS was designed to solve the fundamental problems associated with storing, managing, accessing, securing, and auditing data in traditional file systems.

Traditional database applications were developed on top of the databases, which led to challenges such as data redundancy, isolation, integrity constraints, and difficulty managing data access. A layer of abstraction was required between users or apps and the databases at a physical and logical level.

Introducing DBMS software to manage databases results in the following benefits:

Data security. DBMS allows organizations to enforce policies that enable compliance and security. The databases are available for appropriate users according to organizational policies. The DBMS system is also responsible to maintain optimum performance of querying operations while ensuring the validity, security and consistency of data items updated to a database.

Data sharing. Fast and efficient collaboration between users.

Data access and auditing. Controlled access to databases. Logging associated access activities allows organizations to audit for security and compliance.

Data integration. Instead of operating island of database resources, a single interface is used to manage databases with logical and physical relationships.

Abstraction and independence. Organizations can change the physical schema of database systems without necessitating changes to the logical schema that govern database relationships. As a result, organizations can upgrade storage and scale the infrastructure without impacting database operations. Similarly, changes to the logical schema can be applied without altering the apps and services that access the databases.

Uniform management and administration. A single console interface to perform basic administrative tasks makes the job easier for database admins and IT users.

For data-driven business organizations, DBMS can turn into extremely complex technology solutions that may require dedicated resources and in-house expertise. The size, cost and performance of a DBMS varies with the system architecture and use cases, and should therefore be evaluated accordingly. Also, a DBMS failure can incur significant losses to organizations that fail to maintain optimal functionality of a DBMS system.

Related reading

Original reference image:

Free E-book: The Beginner's Guide to MongoDB

These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing www.bmc.com/blogs.

# data[].attributes.attachment_link.title
DBMS: An Intro to Database Management Systems

# data[].attributes.attachment_link.description
Database is any collection of electronic records that can be processed to produce useful information. The data can be accessed, modified, managed, controlled and organized to perform various data-processing operations. The data is typically indexed across rows, columns and tables that make workload processing and data querying efficient.

# data[].attributes.body
theo cái sườn mình đã ghi trong word gửi thầy

# data[].attributes.attachment_link.readable_text
Increasing the storage capacity might be the easiest and most popular solution to the ever-expanding quantities of data. Just go out and buy more storage. Triple your storage every year by getting a couple of hundred terabytes (or petabytes!). But if you don't know how to maximize your current resources, this approach is like buying a warehouse down the corner and leaving it half empty and disorganized.

This is where Storage Management comes into play!

Storage Management will attempt to maximize your current resources through various techniques, software, or hardware, or processes. In this article, we will dive deep into storage management, what it is, and how to implement it?

1. What is Storage Management?

2. Storage Management Advantages

3. Software and Hardware: SRM and SAN

4. Implementing Storage Management

5. The Future of Storage Management Technology

What is Storage Management?

Storage Management refers to the processes that help make data storage easier through software or techniques. It tries to improve and maximize the efficiency of data storage resources. Storage management processes can deal with local or external storage such as NAS, SAN, USBs, SDDs, HDD, the Cloud, etc.,

Storage management techniques or software can be divided into the following four subsets:

There are a variety of technologies or systems that fall into one or multiple of these subsets, these can be:

Volume Migration

Storage Virtualization

Snapshot and Mirroring

Auto-Provisioning

Process Automation

Disaster and Recovery

And more...

As mentioned before, the goal of storage management is to improve the performance of resources, not to expand capacity.

These techniques and software will develop the ability to store data and secure it properly.

Storage Management Advantages

We already know what is storage management and what are its methodologies, but what are the characteristics that storage management can improve?

What advantages can it bring to the business to the IT department?

Reduce Capital and Operational Expenses:
The most significant expenses when it comes to storage is maintaining and operating the infrastructure.The CapEx can reduce because a business will not have to expand storage capacity that often. OpEx can also be reduced as ongoing operations on storage are decreased.

Makes Management Easier:
Storage management systems can help users save time through automated tasks, centralized consoles, or by logging remotely.It can also reduce the number of IT staff needed to run the storage infrastructure. Storage management can also make virtualized or cloud environments more comfortable to manage from a single location.

Enhance Performance:
One of the main goals of storage management is to improve the performance of the existing storage resources.For example, compressing data can dramatically reduce the amount of storage and improve file transfer speeds. Automatic storage provisioning can reduce the time it takes to provision storage resources.

Speed and Flexibility:
Storage management solutions should be able to work in real-time and adapt to sudden changes in the storage resources. For example, storage replication is a managed service that replicates stored data in real-time. Storage virtualization can also help improve flexibility and reduce wasted storage.Virtualization can create a pool of physical storage from multiple devices into a single logical storage device. Storage capacity can be easily relocated as the business changes needs.

Higher Availability:
This is probably one of the biggest benefits of storage management. For example, technologies such as Replication, Snapshot and Mirroring, Migration, and Disaster and Recovery (DR) can help you have higher availability and reliability on data.All these storage techniques can help backup and restore data fast, but some can also serve as primary storage.

Software and Hardware: SRM and SAN

There are many closely related concepts to store management, such as SRM or SAN. Such concepts also aim to manage the storage, either through software or hardware, but are not exactly considered "Management Processes."

Storage Resource Management vs. Storage Management

SRM is a software solution that aims to optimize the speed and performance of the storage space that is used in the Storage Area Network (SAN).

It uses a variety of methodologies to identify underutilized storage, allocate storage capacity, transfer old data to alternative media, and can predict storage trends and requirements. SRM also helps to manage configurations, policies, storage media, and more.

Although it sounds the same, Storage Resource Management (SRM) is a different concept than storage management.

SRM refers to the specific software, either as standalone or as a part of a software bundle. SRMs are becoming increasingly sophisticated. For example, IntelliMagic Vision for SAN Storage and Fabric is able to coordinate the activities on SAN storage devices and predict space usage through the use of AI techniques.

IntelliMagic offer a free trial.

Some other examples of SRMs are Symantec, DellEMC SRM, Northern Parklife, or DataCore Software.

On the other hand, storage management refers to the whole package as in, equipment, processes, and software.

So, that makes SRM a subset of storage management.

Networked Storage Management vs. Storage Management

Networked Storage Management solutions such as the Storage Area Networks (SAN) or the Network Area Storage (NAS) device are also similar concepts when it comes to managing storage.

A SAN particularly is a computer network used to improve the accessibility of storage resources.

The elements in the SAN could be disk arrays or tapes that are networked together but cannot be managed by a server, it is controlled by a SAN management software.

Most of the times, SAN and NAS used their proprietary software and hardware, so that makes them less flexible.

As mentioned above, SRM aims to improve the speed and efficiency of the storage used in SANs. So with an SRM, you could manage all SANs from a single server.

Implementing Storage Management.

Storage management is a broad concept, that includes techniques, software, processes that aim at improving the performance, availability, recoverability, and capacity of the storage resources.

The first step to implement storage management would be to train IT personnel and storage administrators on best storage management practices.

There are a couple of storage management standards and organizations to start getting information.

An example is the Storage Management Initiative Specification, (SMI-S), which is a data storage standard developed by the Storage Networking Industry Association (SNIA).

Aside from methodologies and processes, storage management can be implemented as software or included in the hardware. A couple of examples about the SRM software as mentioned before are, Symantec, DellEMC SRM, Northern Parklife, or DataCore Software, are good examples.

A NAS can be an excellent example of storage management in hardware.

Where can storage management be implemented?

Storage management processes and software can be implemented in any storage, whether it is primary, backup, or archives.

The Future of Storage Management Technology.

SRMs, NAS, SAN, and DAS (Direct Attached Storage) solutions present a fantastic set of methodologies to improve the performance, availability, or recoverability of your data.

But all of these have one common problem; you cannot manage all your storage resource from a single console. You need one admin console for each piece of storage.

The new concept of Software-Defined Storage (SDS) promises to revolutionize the way we store, manage, and collect data. SDS decouples data storage policy-based provisioning and management from its underlying hardware.

Software-Defined Storage virtualizes NAS, SANs, and DAS hardware as virtual disks. In other words, SDS introduces storage virtualization to separate the storage hardware resources from the management software.

The best thing is that SDS can be implemented in a wide variety of appliances, from generic servers to traditional SANs, NAS, etc. Since the intelligence is decoupled from the hardware, it does not need any proprietary device to run.

The intelligence running on SDSs will provide all the methodologies shown above, such as thin provisioning, snapshot, mirroring, backup and DR, automation, etc.

The benefits of SDS on storage management?

Manage and unify every storage resource as DAS and NAS in your data center.

No dependence on proprietary hardware.

100% virtualized storage with the management interface that comes with countless features.

Manage all storage from a single console, take snapshots, compress, create backups and DR copies, etc.

Summary

The goal of storage management is not precisely to expand capacity. If you want more storage capacity, go out and buy a couple of HDDs. Instead, storage management deals with the existing resources and finds ways to improve and maximize them.

This concept covers everything, from software, hardware, storage techniques, and processes. For example, compressing volumes to save space and increase data transfers, or create snapshots to decrease the size of backups.

Storage management software can address one or various of these techniques. Some tools may deal only with virtualization, while others might be focused on storage provisioning or security. The hardware such as NAS or SAN can have their proprietary management systems, which make them effective but less flexible.

But storage management is evolving.

One of the biggest reasons why enterprises are changing towards a Software-Defined Storage is because they can centralize and manage all resources under a single console.

# data[].attributes.attachment_link.title
Storage Management - What is it and How to Implement & Monitor!

# data[].attributes.attachment_link.description
Increasing the storage capacity might be the easiest and most popular solution to the ever-expanding quantities of data. Just go out and buy more storage. Triple your storage every year by getting a couple of hundred terabytes (or petabytes!).

# data[].attributes.headline
New technology in data storage and management?

# data[].attributes.subject
New technology in data storage and management?

# data[].attributes.body
MySQL, Microsoft SQL server, Oracle, MongoDB, PostgresSQL