What is Federated Learning?

Federated Learning is a cutting-edge approach to machine learning where data remains on local devices, like smartphones, instead of being sent to a central server. Instead of sharing raw data, devices train models locally and only share updates with a central server, which aggregates them to improve the global model. This method enhances privacy, security, and efficiency, as sensitive data never leaves the device. By reading this article, you’ll understand how Federated Learning works, its real-world applications, and its benefits, such as improved data privacy, reduced bandwidth usage, and the ability to create smarter AI models without compromising user data.

This article was published as a part of the Data Science Blogathon.

Understanding about the Federated Learning

This process is repeated, continuously refining the model. Federated Learning is a training technique for central models, utilizing decentralized data sources while ensuring hyper-personalization, minimal latencies, and, most importantly, privacy preservation. This article serves as an introductory guide to understanding the basics of Federated Learning.

Checkout this article about the Machine Learning

Types of Federated Learning

There are mainly three types of federated learning:

• Horizontal Federated Learning: Data samples are distributed across devices or servers, and the model is trained collaboratively.
• Vertical Federated Learning: Features are divided between devices, and the model is trained on complementary features.
• Federated Transfer Learning: Pre-trained models are fine-tuned on decentralized data for specific tasks, reducing the need for extensive local data.

Understanding Federated Learning with an Example

Federated Learning is born at the intersection of on-device AI, blockchain, and edge computing/IoT.

Here we train a centralized Machine Learning model on decentralized data! Let us take a hypothetical problem statement, and understand how federated learning works, step by step.

• Suppose, you got selected as a machine learning intern in a company, and your task is to create a robust machine learning application, that needs to train itself on user-sensitive data.
• You’re allowed to extract user data, aggregate it from many users, and stack them up on a centralized cloud server, for your model to crunch it. You are a smart guy, and you are doing your job!

But wait. Isn’t your work invading someone’s privacy?

• Yes, it is. And that is not an ethical practice in technology.
• You discussed it in a meeting, and your boss is now worried about the next steps. In the meantime, you and your teammates started discussing the matter.
• One of them yelled, “What if we don’t take user-sensitive data, but train our model locally, on each device?”

That was a good idea. But how can we do this?

We will train our model on the devices themselves, and not on the centralized server, that exposes sensitive data!  The local data generated by the user history, on a particular device, will now be used as on-device data to train our model and make it smarter, much quicker.

Let us sum up the plan now!

• So, our centralized machine learning application will have a local copy on all devices, where users can use them according to our need.
• The model will now gradually learn and train itself on the information inputted by the user and become smarter, time to time.
• The devices are then allowed to transfer the training results, from the local copy of the machine learning app, back to the central server.

Remember, only results, not data!

• This same process happens across several devices, that have a local copy of the application. The results will be aggregated together in the centralized server, this time without user data.
• The centralized cloud server now updates its central machine learning model from the aggregated training results, which is now far better than the previously deployed version.
• The development team now updates the model to a newer version, and users update the application with the smarter model, created from their own data!

WOW! That’s pretty awesome!

• All is going pretty good, while one of your teammates seems to be very worried about the device battery power issue!
• She points, “Such long and costly training would always drain our phone battery, guys!”
• So, do we have a solution to this? Yes, of course, we do!
• Don’t worry. Devices will only participate in the training process when users are not using it. This can occur while your phone is on the charge, in do not disturb mode, or idle.

So, what did we learn about Federated Learning?

In a nutshell, Federated Learning with the above 6 steps discussed, will now create a system that encrypts the user-sensitive data with an encryption key that is not in the hands of your centralized cloud server.

Such an approach is referred to as the Secure Aggregation Principle, where our server is allowed to secure and combine the encrypted results and decrypt only the aggregated results.

This kind of functional encryption is simply said to be a zero-sum masking protocol. Zero-sum masks sum to 1 in one direction, and 0 in another. One of them combines and secures the encrypted or secure user data, while the next decrypts the training results to the server.

This process continues until completed, and then the masks cancel out each other.

That is all about Federated Learning!

Read More about the Deep Learning in this article!

How Does Federated Learning Works?

Federated learning works by training a central model across decentralized devices or servers through an iterative optimization process. Instead of moving all data to a central location, the model is trained locally on each device, and only the model parameter updates or gradients are shared with a central server. This allows the central model to be optimized over a large number of iterations by aggregating updates from many devices while maintaining data privacy. However, this decentralized training scenario also introduces challenges like dealing with heterogeneity in the data distributions across devices, ensuring convergence despite limited communication rounds, and developing efficient techniques for aggregating model updates from a massive number of sources. Advanced optimization algorithms tailored for the federated setting are an active area of research with Federated Learning with Keras and TensorFlow

Before we start, please make sure that your environment is correctly set up and import all the dependencies.

Hands-on with Federated Learning!

Setting Up the Environment and Dependencies

Before diving into the Federated Learning implementation, ensure that your environment is correctly configured. Run the following commands to uninstall unnecessary packages and install the required dependencies:

!pip uninstall --yes tensorboard tb-nightly 
!pip install --quiet --upgrade tensorflow-federated-nightly 
!pip install --quiet --upgrade nest-asyncio 
!pip install --quiet --upgrade tb-nightly # or tensorboard, but not both
import nest_asyncio 
nest_asyncio.apply()

Importing Necessary Libraries

Next, import the required libraries for data preprocessing, model creation, and training:

import collections 
import numpy as np 
import tensorflow as tf 
import tensorflow_federated as tff 
np.random.seed(0)

Loading and Preprocessing Data

For this example, we will use the federated version of the MNIST dataset. The preprocess function is defined to format the data, and the make_federated_data function prepares the federated training data:

def preprocess(dataset): 
    def batch_format_fn(element): 
        return collections.OrderedDict( 
            x=tf.reshape(element['pixels'], [-1, 784]), 
            y=tf.reshape(element['label'], [-1, 1]) 
        ) 
    return dataset.repeat(NUM_EPOCHS) \ 
        .shuffle(SHUFFLE_BUFFER) \ 
        .batch(BATCH_SIZE) \ 
        .map(batch_format_fn) \ 
        .prefetch(PREFETCH_BUFFER)

def make_federated_data(client_data, client_ids): 
    return [preprocess(client_data.create_tf_dataset_for_client(x)) for x in client_ids]

sample_clients = emnist_train.client_ids[:NUM_CLIENTS] 
federated_train_data = make_federated_data(emnist_train, sample_clients) 

print('Number of client datasets:', len(federated_train_data)) 
print('First dataset:', federated_train_data[0])

Creating the Model

Now, we define the model architecture using Keras. The create_keras_model function sets up a simple neural network, and the model_fn function converts it into a TFF model:

def create_keras_model(): 
    return tf.keras.models.Sequential([ 
        tf.keras.layers.InputLayer(input_shape=(784,)), 
        tf.keras.layers.Dense(10, kernel_initializer='zeros'), 
        tf.keras.layers.Softmax(), 
    ])

def model_fn(): 
    keras_model = create_keras_model() 
    return tff.learning.from_keras_model( 
        keras_model, 
        input_spec=preprocessed_example_dataset.element_spec, 
        loss=tf.keras.losses.SparseCategoricalCrossentropy(), 
        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()] 
    )

Training the Model on Decentralized Data

The following code snippet demonstrates the training process on decentralized or user data. The iterative_process is used to iteratively train the model on the federated training data:

state, metrics = iterative_process.next(state, federated_train_data) 
print('round 1, metrics:', metrics) 

NUM_ROUNDS = 11 
for round_num in range(2, NUM_ROUNDS): 
    state, metrics = iterative_process.next(state, federated_train_data) 
    print('round {:2d}, metrics:{}'.format(round_num, metrics))

Evaluating and Testing the Model

Finally, we evaluate and test the trained model using the tff.learning.build_federated_evaluation function:

evaluation = tff.learning.build_federated_evaluation(MnistModel) 
train_metrics = evaluation(state.model, federated_train_data) 

federated_test_data = make_federated_data(emnist_test, sample_clients) 
test_metrics = evaluation(state.model, federated_test_data)

Conclusion

Federated Learning seems to have a lot of potential. Not only does it secure user-sensitive personal data, but it also aggregates results and identifies common patterns from a lot of users, which makes the global model robust, day by day.

It trains itself as per its user data on mobile phones or edge devices, keeps the personal data secure, and then comes back as a smarter model, which is again ready to test itself from its user’s local data! Training and testing became smarter and more privacy-preserving.

Be it training, testing, or information privacy, Federated Learning created a new era of secured AI that can leverage large decentralized datasets without directly accessing the raw data.

Federated Learning is still in its early stages and faces numerous challenges with its design and deployment, especially for complex deep learning and generative AI models. A good way to tackle this challenge is by defining the Federated Learning problem and designing a data pipeline such that it can be properly productized.

You can run a TensorFlow tutorial of Federated Learning here to get your hands on it!

Master all the important concepts of data science and machine learning with our AI/ML Blackbelt Plus Program.

Key Takeaways

• Federated learning enables training machine learning models on decentralized data across devices without sharing raw data, preserving privacy.
• Instead of centralized data aggregation, the model is trained locally on each device, and only model updates are sent to a central server.
• It leverages secure aggregation techniques like encryption to protect user data during the aggregation process.
• Federated learning is advantageous for scenarios where data cannot leave the source due to privacy regulations or constraints.
• Frameworks like TensorFlow Federated facilitate the development and deployment of federated learning systems.

Frequently Asked Questions

References:

• Federated Learning: Strategies for Improving Communication Efficiency — Google Research
• Federated Learning: A Survey on Enabling Technologies, Protocols, and Applications -ResearchGate
• TensorFlow Federated Tutorials!

Sukanya Bag

An ace multi-skilled programmer whose major area of work and interest lies in Software Development, Data Science, and Machine Learning. A proactive and detail-oriented individual who loves data storytelling, and is curious and passionate to solve complex value-oriented business problems with Data Science and Machine Learning to deliver robust machine learning pipelines that ensure maximum impact.

In my free time, I focus on creating Data Science and AI/ML content, providing 1:1 mentorships, career guidance and interview preparation tips, with a sole focus on teaching complex topics the easier way, to help people make a successful career transition to Data Science with the right skillset!