Introduction
Before analysing a dataset, you need to know what is in it. How many rows and columns are there? Which columns hold numbers you can average, and which hold labels you can only count? Are any values missing? Mistakes at this stage spread through every later step: averaging a column of postcodes or sorting sizes alphabetically gives answers that look precise and mean nothing. This lesson is about describing a table before using it.
How it works
A dataset is usually a table: rows are observations, columns are variables. In plain Python a convenient form is a list of dictionaries:
rows = [
{"city": "Dhaka", "population_m": 10.2, "coastal": False},
{"city": "Sylhet", "population_m": 0.5, "coastal": False},
]
Its shape is the number of rows, len(rows), and the number of columns, len(rows[0]). The list of columns with their types is the table's schema.
Variables come in a few kinds, and each kind allows different operations:
- Numeric variables hold numbers. Continuous ones, like height or price, can take any value; discrete ones, like the number of children, are counts. You can add, average and compare them.
- Categorical variables hold labels: a city, a colour, a product code. You can count them and compare them for equality, but averaging them makes no sense, even when the labels look like numbers, such as postcodes or phone numbers.
- Ordinal variables are categories with an order: sizes, ratings, education levels. You can sort them and find a median, but only with an order you write down; Python's alphabetical order would put "large" before "medium".
- Boolean (yes/no) variables have two values and are often counted as proportions.
- Text and dates need their own handling, covered later.
Data read from a CSV file arrives as text: "10.2", not 10.2. Part of preparing data is checking each column's values and converting them. A useful habit is to infer each column's type from its values, as the example does: if every value is digits, it is an integer column; if every value converts to float, it is decimal; otherwise it is text. In pandas, df.dtypes, df.shape and df.info() print the same overview.
Example
# Data types: work out what kind of values each column holds.
rows = [
{"city": "Dhaka", "population_m": "10.2", "rank": "1", "coastal": "no", "size": "large"},
{"city": "Chattogram", "population_m": "5.3", "rank": "2", "coastal": "yes", "size": "large"},
{"city": "Sylhet", "population_m": "0.5", "rank": "5", "coastal": "no", "size": "small"},
{"city": "Khulna", "population_m": "0.7", "rank": "3", "coastal": "no", "size": "medium"},
]
def kind(values):
if all(v.isdigit() for v in values):
return "integer"
try:
for v in values:
float(v)
return "decimal"
except ValueError:
pass
if set(values) <= {"yes", "no"}:
return "yes/no"
return "text"
for column in rows[0]:
values = [row[column] for row in rows]
print(column, "->", kind(values), "with", len(set(values)), "distinct values")
# size is text, but its values have an order: an ordinal category.
order = ["small", "medium", "large"]
by_size = sorted(rows, key=lambda r: order.index(r["size"]))
print([r["city"] for r in by_size])Line by line
rowsholds four cities; every value is text, as if read from a file.kindchecks the values in order: all digits, then all convertible to a number, then only yes and no, and otherwise text.- The loop prints each column's kind and how many different values it has.
sizeis text but ordinal, so sorting uses an explicit order list instead of the alphabet.
Expected result
city -> text with 4 distinct values
population_m -> decimal with 4 distinct values
rank -> integer with 4 distinct values
coastal -> yes/no with 2 distinct values
size -> text with 3 distinct values
['Sylhet', 'Khulna', 'Dhaka', 'Chattogram']
In the real world
Data engineers write schemas for every table so that programs reject a row whose age is "forty" before it reaches an analysis. Survey tools record whether each question's answers are categories, ordinal scales or numbers, because that decides which charts and statistics are valid. A classic error in business reports is a total of customer ids or a mean of postcodes, numbers that are really labels.
Common mistakes
- Averaging labels that look like numbers, such as postcodes, ids or phone numbers.
- Sorting ordinal categories alphabetically, which puts "high" before "low" and "medium".
- Forgetting that values read from files are text, so
"9" > "10"is True. - Assuming every row has the same columns without checking.
Best practices
- Inspect the shape, the types and a few rows of every new table before analysing it.
- Write the order of every ordinal variable explicitly in your code.
- Convert columns once, early, and keep the original text if conversion fails.
Summary
A table holds observations in rows and variables in columns, and in Python can be a list of dictionaries. Variables are numeric (continuous or discrete), categorical (nominal or ordinal), boolean, text or dates, and each kind allows different operations. Values from files are text, so check and convert each column, and give ordinal variables an explicit order.
Quick check
Key takeaways
- Every column has a kind, and the kind decides which summaries and operations make sense.
- Values read from files are text until you convert them; check and convert each column.
- Ordinal categories need their order written down explicitly; alphabetical order is usually wrong.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.