Data Science 4 - Definition of Big Data

-

 Definition of Big Data:


Definition by Ernst and Young: "Big Data refers to the dynamic, large and disparate volumes of data created by people, tools, and machines."

Technology Requirement: Demands new, innovative, and scalable technology for collection, hosting, and analytical processing.

Purpose: Derive real-time business insights related to consumers, risk, profit, performance, productivity management, and enhanced shareholder value.

II. Characteristics of Big Data - The V's:


Velocity:


Definition: Speed at which data accumulates.

Data Generation: Occurs extremely fast, with near or real-time streaming, local, and cloud-based technologies facilitating quick processing.

Volume:


Definition: Scale of data or the increase in stored data.

Drivers: Increase in data sources, higher resolution sensors, and scalable infrastructure.

Example: World population using digital devices generates approximately 2.5 quintillion bytes of data every day.

Variety:


Definition: Diversity of data, including structured and unstructured forms.

Data Sources: From various machines, people, processes, and internal/external organizations.

Example: Different types of data, such as text, pictures, film, sound, health data from wearables, and IoT-connected devices.

Veracity:


Definition: Quality, origin, conformity to facts, and accuracy of data.

Attributes: Consistency, completeness, integrity, and ambiguity.

Example: 80% of data considered unstructured; challenges in categorization, analysis, and visualization.

Value:


Definition: Ability and need to turn data into value, not limited to profit.

Examples: Medical or social benefits, customer satisfaction, employee satisfaction.

III. Examples of V's in Action:


Velocity Example: Every 60 seconds, hours of footage uploaded to YouTube, showcasing the rapid accumulation of data.

Volume Example: Global population using digital devices generates 2.5 quintillion bytes of data daily, equivalent to 10 million Blu-ray DVDs.

Variety Example: Diverse types of data, including text, pictures, film, sound, health data, and IoT data.

Veracity Example: Devising ways to produce reliable insights from 80% unstructured data.

IV. Big Data Analysis and Tools:


Scale Challenges: Conventional data analysis tools not feasible due to the scale of data.

Alternative Tools: Apache Spark, Hadoop, and its ecosystem leverage distributed computing power to extract, load, analyze, and process data.

Benefits: Provide new insights, knowledge, and ways for organizations to connect with customers and enhance services.

V. Conclusion:


Data Journey: From smartwatches, smartphones, and workout tracking devices, user data embarks on a global journey through big data analysis, returning with valuable insights.




Comments

Post a Comment

Popular posts from this blog

Lila's Journey to Becoming a Data Scientist: Her Working Approach on the First Task

-
  Education and Skill Acquisition: Lila, with an economics background, decides to transition to data science. Enrolls in the IBM Data Science Professional Certificate for comprehensive training. Gains proficiency in statistics, machine learning, and programming languages. Building a Strong Foundation: Develops a deep understanding of data manipulation and visualization using Python libraries. Visualization for Storytelling: Learns to create informative visualizations, aiding in effective communication of findings. Hands-On Experience: Engages in Kaggle competitions and personal data projects for practical experience. Establishes a GitHub profile to showcase her projects and skills. Data Wrangling and Preprocessing: Focuses on data cleaning, preprocessing, and handling missing data and outliers. Communication and Storytelling: Hones data storytelling skills using tools like Matplotlib and Plotly. Networking and Collaboration: Actively participates in data science communities, attend...
Read more

Notes on Hiring for Data Science Teams

-
  Unicorn Myth in Hiring: Companies often seek individuals with a myriad of skills for data science roles. Ideal candidates possess domain-specific knowledge, excel in data analysis, and have strong presentation and storytelling skills. However, finding such a "unicorn" is rare, and the focus should shift to other essential qualities. Resonance with Company DNA: The key is to identify applicants who resonate with the company's values and business domain. While analytics skills can be taught, passion for the business is intrinsic and contributes significantly to productivity. Curiosity Matters: Curiosity is a top priority in hiring for a data science team. Candidates should demonstrate a broad curiosity, showing interest in various aspects of their surroundings beyond just data science. Sense of Humor: A lighthearted approach is crucial, as being too serious may hinder collaboration and creativity. A sense of humor aids in navigating challenges and fosters a positive work ...
Read more

switch functions

-
  Nested IF Function : The  nested IF  function allows you to perform tests with multiple conditions and return different values based on the results of those tests. It is called “nested” because one IF function is embedded inside another IF function. Syntax of a generic nested IF formula: =IF(condition1, result1, IF(condition2, result2, IF(condition3, result3, result4) ) ) Each subsequent IF function is embedded into the  value_if_false  argument of the previous function. SWITCH Function : The  SWITCH  function is a more compact alternative to nested IF statements. It compares an expression against a list of values and returns the result according to the first matching value. Syntax of the SWITCH function: =SWITCH(expression, value1, result1, [default or value2, result2], ... [default or valueN, resultN]) It has four arguments: expression : The value to compare against the list of values. valueN : A value com...
Read more