What data science is

Intermediate

Introduction

Every organisation now collects data: shops record sales, hospitals record treatments, schools record results, apps record every tap. Data science is the work of turning that data into answers: which product sells best in winter, which students need help, whether a new design made users stay longer. It combines three skills: programming to handle the data, statistics to reason about it, and knowledge of the subject to ask the right questions and understand the answers.

This course teaches the whole process with plain Python and its standard library. Professional data scientists usually use libraries such as pandas and NumPy, and the lessons show what the same steps look like there, but doing them by hand first means you will understand exactly what those libraries do for you.

How it works

A data science project follows a cycle of steps:

  1. Ask a precise question. "Is studying useful?" is too vague; "do students who study 4 or more hours a week score higher on the final test than those who study less?" can be answered with data.
  2. Collect the data that can answer it: from files, databases, surveys or measurements.
  3. Clean it: fix or remove missing, duplicated and impossible values.
  4. Explore it: count, average, group and compare, and look for patterns and surprises.
  5. Model it: describe the relationship with a formula or a rule that can predict new cases.
  6. Communicate the result: the answer, the numbers behind it and how sure you are.

In practice the steps repeat. Exploring often shows that the data needs more cleaning, and a result often raises a better question.

Data is usually organised as a table. Each row is an observation, such as one student, and each column is a variable, such as hours studied or test score. In plain Python, a natural way to hold a table is a list of dictionaries, one dictionary per row, with the column names as keys.

The simplest useful analysis compares groups: split the rows by some condition and compare an average. It is a good first step, but be careful with what it means. If students who study more score higher, that is a link, a correlation; it does not prove that studying caused the higher scores, because other differences between the groups might explain it.

Example

# A question, some data and an answer: the data science process in miniature.
# Question: do students who study more hours score higher?
students = [
    {"name": "Asha", "hours": 2, "score": 58},
    {"name": "Ben", "hours": 5, "score": 74},
    {"name": "Chen", "hours": 1, "score": 49},
    {"name": "Dina", "hours": 6, "score": 81},
    {"name": "Eli", "hours": 4, "score": 70},
    {"name": "Farah", "hours": 3, "score": 66},
]

# Explore: split the students into two groups and compare their average scores.
more = [s["score"] for s in students if s["hours"] >= 4]
less = [s["score"] for s in students if s["hours"] < 4]
avg_more = sum(more) / len(more)
avg_less = sum(less) / len(less)

# Communicate: say what was found, with the numbers behind it.
print("4 or more hours:", len(more), "students, average score", round(avg_more, 1))
print("under 4 hours:", len(less), "students, average score", round(avg_less, 1))
print("difference:", round(avg_more - avg_less, 1), "points")
Open in compiler

Line by line

  1. The comment states the question first, so everything after it has a purpose.
  2. students is the dataset: six observations, each with three variables.
  3. Two list comprehensions split the scores into two groups by hours studied.
  4. Each group's average is its total divided by its size.
  5. The output gives the size and average of each group and the difference, rounded so it is easy to read.

Expected result

4 or more hours: 3 students, average score 75.0
under 4 hours: 3 students, average score 57.7
difference: 17.3 points

In the real world

Supermarkets compare sales before and after a promotion, streaming services compare how long people watch with and without a new feature, and health researchers compare outcomes between treatments. Large organisations run thousands of such comparisons; the best ones are planned from a precise question, use enough data, and report their uncertainty along with the answer.

Common mistakes

  • Starting with the data instead of a question, which produces many numbers and no answer.
  • Treating a link between two variables as proof that one causes the other.
  • Comparing groups of very different sizes without saying so; an average of two people is fragile.
  • Reporting a single number with no context, such as how many observations it is based on.

Best practices

  • Write the question down before you look at the data.
  • Always report how many observations each number is based on.
  • Keep the raw data unchanged, and do cleaning and analysis in code that can be run again.

Summary

Data science turns data into answers through a cycle of steps: ask a precise question, collect the data, clean it, explore it, model it and communicate the result. Data is held as a table of observations and variables, and even a simple comparison of group averages can answer a question, as long as you report its size and remember that a link is not a cause.

Quick check

1. Which question can data answer directly?
2. In a table of students, what is one row?
3. Students who study more score higher. What does that show on its own?
4. Why report the number of observations behind an average?

Key takeaways

  • Data science starts with a precise question and ends with an answer someone can act on.
  • The steps are: ask, collect, clean, explore, model and communicate, often repeated.
  • A comparison of group averages is the simplest useful analysis, but it shows a link, not a cause.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.