Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Gen AI Powered Data Insight Generation using LIDA 

Ankit Bajaj
Ankit Bajaj Last Updated : 18 Jun, 2024
6 min read

Introduction

This article will introduce the readers to LIDA, an open-source python library for generating detailed data visualizations and appealing infographics. We will first understand that how LIDA works and what are its core capabilities and then finally see it in action by building a Streamlit application that will enable the user to explore the provided csv dataset and unearth valuable insights automatically by creating amazing data visualizations.

Learning Objectives

  • Understand the challenges associated with manual data exploration and analysis.
  • Explore LIDA architecture and core building blocks.
  • Learn to build a fully functional Streamlit application for automated data exploration and insight generation.

This article was published as a part of the Data Science Blogathon.

Table of contents

Challenges with Manual Data Exploration

Manual data exploration is a labor intensive process that demands significant time and effort to clean, analyze, and visualize data. Analysts often face the challenge of sifting through large datasets, which
increases the likelihood of human error and overlooked patterns or insights. Additionally, the manual approach can be inconsistent, as it relies heavily on the individual skills and expertise of the analyst, making it difficult to reproduce results or scale the process for larger datasets.

Automating data exploration accelerates the analysis process, ensuring more accurate and comprehensive insights. Automation tools, like LIDA, streamline data visualization, and insight generation, allowing users to focus on decision-making and strategic planning.

What is LIDA and How Does it Work?

LIDA is a open-source python library for generating data visualizations and infographics. LIDA is grammar agnostic and can work with any programming language. It also supports multiple visualization libraries like matplotlib, seaborn etc.

LIDA Architecture

LIDA consists of the following 4 key modules that work together in a sequence to generate automatic data visualizations and infographics:

Summarizer

  • Function: Converts datasets into a rich but compact natural language representation (context)
  • Process: Uses rules and large language models (LLMs) to analyze the dataset.

Goal Explorer

  • Function: Generates a set of potential “goals” based on the dataset context.
  • Process: Utilizes LLMs to interpret the context and suggest relevant visualization goals.

Viz Generator

  • Function: Generates, evaluates, repairs, filters, and executes visualization code to meet specified goals.
  • Process: Leverages LLMs to create visualization code in the appropriate programming language or grammar.

Infographer

  • Function: Generates stylized infographics based on the visualization and style prompts.
  • Process: Applies image generation models (IGMs) to transform visualizations into styled infographics.

Now that we are familiar with the building blocks of LIDA and their respective functions, let’s understand that how all these blocks integrate and work together in a single workflow:

  • Dataset Input: The user provides a CSV dataset (e.g., Cars.csv).
  • Summarization: The Summarizer processes the dataset and generates a natural language context.
  • Goal Exploration: The Goal Explorer uses the context to suggest possible visualization goals.
  • Visualization Generation: The Viz Generator creates and executes code to produce visualizations based on the selected goals.
  • Infographic Creation: The Infographer transforms these visualizations into styled infographics according to user-defined prompts.
  • Output Delivery: The system outputs a natural language summary, suggested goals, visualization code, and the final stylized infographics.

This integrated approach streamlines the process of data exploration, visualization, and infographic creation, making it efficient and user-friendly.

Building Application for Automatic Insight Generation

Now that we have a pretty fair idea of LIDA and it’s functioning, let’s roll up our sleeves and get into some action by building a Streamlit application that will accept a CSV dataset as input and then leverage LIDA to generate automatic data visualizations

Step1: Install Python Libraries

First things first, let’s install the required python libraries for our application. We will create a requirements.txt file with the following set of libraries:

Python Library Description/Use case
uvicorn A lightning-fast ASGI server for running Python web applications
streamlit An open-source app framework for creating and sharing beautiful, custom web apps
pandas A powerful data manipulation and analysis library providing data structures like DataFrames
lida A toolkit for generating data visualizations and data-faithful infographics
python-dotenv A toolkit for generating data visualizations and data-faithful infographics, compatible with various programming languages and visualization libraries

Then install all the libraries by running the command “pip install -r requirements.txt”

Step2: Integrating LIDA with LLM

Next, we need to integrate LIDA with a LLM that will be used to summarize the dataset, create goals and then finally generate and execute visualization code. LIDA is highly flexible and integrates smoothly with multiple large language model providers, including OpenAI, Azure OpenAI, PaLM, Cohere, and Huggingface. However, for our application, we will be using the GPT-3.5 Turbo model by OpenAI and for that we would need an Open AI API key.

To generate an API key, first, create an OpenAI account or sign in. Next, navigate to the API key page and “Create new secret key”, optionally naming the key. Make sure to save this somewhere safe and do not share it with anyone.

Once we have API key, create a .env file and save your API key over there

Step3: Streamlit Application Logic

Finally, we will create the app.py file containing the Streamlit application logic and LIDA API call. 

import streamlit as st
import pandas as pd
from lida import Manager, TextGenerationConfig , llm  
from PIL import Image
from io import BytesIO
import base64
from dotenv import load_dotenv
import os
import openai

# Configuring the OpenAI API Key
load_dotenv()
openai.api_key = os.getenv('OPENAI_API_KEY')

# To convert charts into images, so that they can be displayed on Stremlit front-end
def base64_to_image(base64_string):
    # Decode the base64 string
    byte_data = base64.b64decode(base64_string)
    # Use BytesIO to convert the byte data to image
    return Image.open(BytesIO(byte_data))

# Streamlit App Code
st.set_page_config(
    page_title="Automatic Insights and Visualization App",
    page_icon="🤖",
    layout="centered",
    initial_sidebar_state="expanded",
)

st.header("Automatic Insights and Visualization 🤖")


menu = st.sidebar.selectbox("Choose an Option", ["Automatic Insights"])

if menu == "Automatic Insights":
    st.subheader("Generate Automatic Insights")
    # Upload CSV dataset as input
    uploaded_file = st.file_uploader("Choose a csv file")
    if uploaded_file is not None:
        dataframe = pd.read_csv(uploaded_file)
        st.write(dataframe)
        btn = st.button("Generate Suggestions", type = "primary")

        if btn: 
            # Generate goals using LIDA
            lida = Manager(text_gen = llm("openai"))
            textgen_config = TextGenerationConfig(n=1, 
                                                  temperature=0.5, 
                                                  model="gpt-3.5-turbo-0301", 
                                                  use_cache=True)
            summary = lida.summarize(dataframe, 
                      summary_method="default", 
                      textgen_config=textgen_config)  
            goals = lida.goals(summary, n=5, textgen_config=textgen_config)

            i = 0
            library = "seaborn"
            imgs = []
            textgen_config = TextGenerationConfig(n=1, temperature=0.2, use_cache=True)
            # Create the corresponding data visualization for each goal
            for i in range(len(goals)):
                charts = lida.visualize(summary=summary, 
                                        goal=goals[i], 
                                        textgen_config=textgen_config, 
                                        library=library)
                img_base64_string = charts[0].raster
                img = base64_to_image(img_base64_string)
                imgs.append(img)

            tab1, tab2, tab3, tab4, tab5 = st.tabs(
            ["Goal 1", "Goal 2", "Goal 3", "Goal 4", "Goal 5"]
            )

            with tab1:
                st.header("Goal 1")
                goals[0].question
                st.image(imgs[0])

            with tab2:
                st.header("Goal 2")
                goals[1].question
                st.image(imgs[1])

            with tab3:
                st.header("Goal 3")
                goals[2].question
                st.image(imgs[2])
            
            with tab4:
                st.header("Goal 4")
                goals[3].question
                st.image(imgs[3])
            
            with tab5:
                st.header("Goal 5")
                goals[4].question
                st.image(imgs[4])

Once all the files are ready, you can run the streamlit application using the command “streamlit run app.py”

Generate Automatic insights
Lida

Conclusion

We explored the challenges associated with manual data exploration and how tools like LIDA help us streamline the process by providing a flexible and fully automatic solution for data exploration and insight generation We also got an understanding of the LIDA system architecture and its core capabilities. Lastly, we saw LIDA in action by building an automatic insight generation application using Streamlit.

Here is the link for the video depicting the final application and it’s working.

Key Takeaways

  • Whether you’re working with Matplotlib or Seaborn, Python or any other programming language, LIDA fits right into your workflow.
  • Leverage the latest language models to generate intelligent insights and recommendations for your data.
  • No step learning curves here. LIDA is designed to be intuitive and easy to use, so you can focus on the things that matter to you and the business – making data-driven decisions.
  • Automating data exploration accelerates the analysis process, ensuring more accurate and comprehensive insights.

Frequently Asked Questions

Q1. What are the different LLM models supported by LIDA?

A. LIDA supports multiple large language model providers like OpenAI, Azure OpenAI, PaLM, Cohere and Huggingface .

Q2. Is an API key required to work with LIDA?

A. LIDA is an open-source library and doesn’t require an API key as you need to install it on your system and run it locally, but you might need an API key for the LLM model that you will be using with LIDA. For example, you will need an OpenAI API key if you are using a model like GPT3.5-Turbo.

Q3. Does LIDA support query-based visualization generation?

A. Instead of relying on LIDA for goal generation, a user can explicitly provide the query/goal and generate the desired chart. LIDA also provides the support for multi-lingual input.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Ankit Bajaj
Ankit Bajaj

As a dedicated Data Analyst with three years of experience, I am passionate about harnessing the power of data to drive informed business decisions. My expertise spans a diverse array of tools and technologies, including Google Analytics (GA), Google Tag Manager (GTM), Python, SQL, Power BI, and the Azure cloud platform. These skills enable me to effectively collect, analyze, and interpret complex data sets, transforming raw data into actionable insights.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details