Federated Learning

Federated learning trains a single shared model across many decentralized devices or organizations, each holding private local data — without that raw data ever leaving its original location or being centralized anywhere.

The Core Motivation: Privacy and Data Locality

Many valuable datasets can't be centralized for legal, privacy, or practical reasons — medical records across different hospitals, personal data on millions of individual phones, sensitive financial data across different institutions. Federated learning lets a model still learn from all of this collective data, without any single party ever needing to see another party's raw data directly.

The Standard Algorithm — FedAvg

1. A central server sends the current global model to a selected subset of participating clients (devices/organizations).
2. Each client trains the model locally on its own private data for a few steps, producing a locally-updated model.
3. Clients send back only their updated model weights (never their raw data) to the central server.
4. The server aggregates these updates — typically by simple averaging — to produce a new, improved global model.
5. Repeat, for many rounds.

The Aggregation Formula

\(\mathbf{w}_k\) is client \(k\)'s locally updated weights; \(n_k\) is that client's local dataset size; \(n\) is the total data size across all participating clients this round. This is a weighted average — clients with more local data contribute proportionally more to the new global model.

Diagram

Code — A Simplified FedAvg Round

import copy

def federated_averaging(global_model, client_datasets, local_epochs=1):
    client_weights = []
    client_sizes = []

    for client_data in client_datasets:
        local_model = copy.deepcopy(global_model)
        train_locally(local_model, client_data, epochs=local_epochs)   # each client trains on its OWN data
        client_weights.append(local_model.state_dict())
        client_sizes.append(len(client_data))

    total_size = sum(client_sizes)
    new_global_weights = {}
    for key in client_weights[0]:
        new_global_weights[key] = sum(
            (client_sizes[i] / total_size) * client_weights[i][key] for i in range(len(client_weights))
        )
    global_model.load_state_dict(new_global_weights)
    return global_model

Common Mistakes

• Assuming federated learning is inherently perfectly private — while raw data never leaves clients, model updates themselves can sometimes leak information about the underlying data under certain attacks, which is why additional privacy techniques (like differential privacy) are often layered on top in sensitive applications.
• Assuming FedAvg's simple weighted average always converges as smoothly as centralized training — highly non-uniform (non-IID) data distributions across clients can make federated training meaningfully harder to converge than standard centralized training on the same combined data.

Interview Relevance

Q: "What specifically gets sent between clients and the server in federated learning, and why does this preserve privacy?" Only model updates (weights or gradients), never raw data — each client trains locally on its own private data and sends back only the resulting model parameters, which the server aggregates (typically via weighted averaging). Since raw data never leaves the client, this directly enables training on collectively valuable data that couldn't legally or practically be centralized.

Practice Question

Why does FedAvg's aggregation formula weight each client's contribution by its local dataset size \(n_k\), rather than averaging all clients equally?

Related DL Notes

• Continual Learning
• Distributed Training
• Data Drift