By the end of this lesson, you will be able to identify structured, semi-structured, and unstructured data types and understand how they differ in storage and analysis requirements.
What it is
Data classification is based on how organized the information is. Structured data fits neatly into rows and columns with predefined schemas, like a spreadsheet or SQL table. Semi-structured data lacks a rigid tabular structure but contains tags or markers to separate elements, such as JSON, XML, or CSV files. Unstructured data has no predefined format or organization, including text documents, images, videos, and audio files.
Related terms include schema-on-read (common for semi/unstructured) versus schema-on-write (common for structured), and NoSQL databases which often handle semi-structured data efficiently.
Why it matters
- Storage Efficiency: Choosing the right database type saves space and cost.
- Query Performance: Structured data allows fast indexing; unstructured requires different search techniques.
- Analysis Tools: SQL works best for structured data, while Python libraries like Pandas or specialized NLP tools are needed for others.
- Data Integration: Understanding formats helps when merging datasets from diverse sources like APIs and logs.
Syntax or steps
To analyze these types, you typically load them into a dataframe using specific parsers. For structured data, use standard CSV readers. For semi-structured JSON, use nested flattening functions. Unstructured text usually requires tokenization before analysis.
Example
import pandas as pd
import json
# 1. Structured Data (CSV-like dictionary)
structured_data = {
"id": [1, 2],
"name": ["Alice", "Bob"],
"age": [30, 25]
}
df_structured = pd.DataFrame(structured_data)
# 2. Semi-Structured Data (JSON string)
json_string = '{"user": {"name": "Charlie", "scores": [90, 85]}, "active": true}'
data_semi = json.loads(json_string)
# Flatten nested JSON for tabular view
df_semi = pd.json_normalize(data_semi)
# 3. Unstructured Data (Raw Text)
text_data = "The quick brown fox jumps over the lazy dog."
# Simple word count (basic analysis step)
word_count = len(text_data.split())
print("Structured:\n", df_structured.head())
print("\nSemi-Structured:\n", df_semi)
print(f"\nUnstructured Word Count: {word_count}")This code demonstrates loading three distinct formats. The structured data maps directly to columns. The semi-structured JSON requires normalization to flatten nested objects into columns. The unstructured text is processed via simple splitting, illustrating that basic metrics can still be derived without a schema.
Common mistakes
- Forcing Unstructured into Tables: Trying to store raw video files in SQL BLOBs without metadata makes retrieval difficult. Use object storage instead.
- Ignoring Nested Structures: Loading JSON without flattening leads to complex queries. Always normalize semi-structured data if tabular analysis is needed.
- Assuming All CSVs are Structured: Poorly formatted CSVs with inconsistent delimiters behave more like semi-structured data and require cleaning.
- Lack of Metadata: Unstructured data needs context (tags, timestamps). Without it, it becomes "dark data" that is hard to find.
When to use it
| Type | Best For | Typical Storage |
|---|---|---|
| Structured | Financial records, inventory, user profiles | Relational DBs (PostgreSQL) |
| Semi-Structured | Web logs, API responses, config files | NoSQL (MongoDB), Data Lakes |
| Unstructured | Email bodies, social media posts, images | Object Storage (S3), Search Engines |
Practice
Guided Exercise: Take a small JSON file containing an array of user objects with nested address fields. Use pd.json_normalize() to convert it into a flat DataFrame.
Challenge: Write a script that reads a plain text log file and extracts all email addresses using regular expressions. This simulates deriving structured insights from unstructured data.
Quick check
Question: Which data type typically requires schema-on-read processing?
Answer: Semi-structured and unstructured data, because their format may vary or lack predefined columns until analyzed.
Summary
Recognizing whether data is structured, semi-structured, or unstructured dictates your tooling and storage strategy. Structured data offers speed and consistency, while semi-structured provides flexibility, and unstructured holds rich, complex information requiring advanced processing techniques.
