Building a Real-time Short News App using HuggingFace Transformers and Streamlit

import streamlit as st
from transformers import pipeline
import json
import requests
from newspaper import Article

Before we code our app we need to get an API to fetch news links based on our search from the internet. As discussed earlier we will be using Free news API to get that link. So for this, we get an API from this link. Follow the steps given in the link to get the API. If you have doubts please comment below so that I can clear your doubts.

Now that we have installed and imported all necessary libraries and also got our API key, we code our Streamlit app in the following way. The final code looks like below –

import streamlit as st
import json
import requests
from newspaper import Article
from transformers import pipeline
st.set_page_config(page_title='Short News App', layout='wide', initial_sidebar_state = 'expanded')
st.title('Welcome to Short News App n Tired of reading long articles? This app summarizes news articles for you and gives you short crispy to the point news based on your search n (This is a demo app and hence is deployed on a platform with limited computational resources. Hence the number of articles  this app can fetch is limited to 5)')
summarizer = pipeline("summarization")
article_titles = []
article_texts = []
article_summaries = []
def run():
    with st.sidebar.form(key='form1'):
        search = st.text_input('Search your favorite topic:')
        submitted = st.form_submit_button("Submit")
        if submitted:
            try:
                url = "https://free-news.p.rapidapi.com/v1/search"
                querystring = {"q":search, "lang":"en", "page":1, "page_size":5}
                headers = {'x-rapidapi-host':"free-news.p.rapidapi.com", 'x-rapidapi-key':"your_api_key"}
                response = requests.request("GET", url, headers=headers, params = querystring)
                response_dict = json.loads(response.text)
                links = [response_dict['articles'][i]['link'] for i in range(len(response_dict['articles']))]
                for link in links:
                    news_article = Article(link, language = 'en')
                    news_article.download()
                    news_article.parse()
                    article_titles.append(news_article.title)
                    article_texts.append(news_article.text)
                for text in article_texts:
                    article_summaries.append(summarizer(text)[0]['summary_text'])
            except:
                print("Try with new search")
    for i in range(len(article_texts)):
        st.header(article_titles[i])
        st.subheader('Summary of Article')
        st.markdown(article_summaries[i])
        with st.expander('Full Article'):
            st.markdown(article_texts[i])
if __name__=='__main__':
    run()

Explanation of the above code

• First, we import all necessary libraries.
• Then we set the page configuration of our app using st.set_page_config(). We give all parameters as shown in the above code.
• Next, we give a title of our app using st. title() as shown in the code. You can give your title and explanation as shown in the above code.
• Now we define three empty lists to store our article titles, article texts, and their corresponding summaries for final display. Next, we create a sidebar in our app for searching the desired topic. We create this search form using st.sidebar().form() of Streamlit as shown in the above code and also we create a submit button using st.form_submit_button().
• So now if the user enters a search term and clicks submit button we have to fetch news articles and display their summaries. We do that as follows –
• First, we get our API URL.
• We give our query parameters like search term, language, number of pages, and number of articles per page.
• Create headers for API as shown code.
• Using requests library we fetch data from the internet using our query parameters and headers.
• We convert the request object from the above step into a dictionary using JSON.
• Next, we extract the links from the dictionary metadata we obtained from the internet using requests.
• For each link we have, we use the newpaper3k library to get the article title and article text and append them into the corresponding empty lists we defined earlier.
• Now that we have article titles, article texts we get our summaries for each article using transformers summarization pipeline as shown in the above code.
• Now we have article titles, article texts, and their corresponding summaries. So we display the article using st.header() and we display the summary of the article using st.markdown(). Before that we create a heading ‘Summary of Article’ and then we display the news summary under this.
• Finally, if the user wants to read the full article, we also give an option to look at the full article using st.expander(). This expander widget of streamlit hides the content in it and displays the content only when the user clicks to expand.

Testing

We built our app. Save the code in a .py file and open the terminal.

Type streamlit run your_app_name.py in the terminal and press enter.

You will see a new tab in your browser with your app.

Deploying

We built and tested our app. It’s time to deploy it. We will deploy this on Streamlit cloud. Streamlit offers to host three apps for free on its cloud.

Go to this link of Streamlit sharing and create an account if you don’t have one already. After you create an account and sign in you will see a web page with the option ‘New App’. Click on it.

It will ask you to connect to a Github repository. So create a Github repository of your app and upload the .py file we just created. We also need a requirements.txt. So create a file with the name ‘requirements.txt’ and type the following in that file,

streamlit
transformers
tensorflow
requests
newspaper3k

save the file and commit changes to your repository. Now come to the Streamlit sharing website and connect your newly created Github repository and click deploy. Streamlit does the rest of the work for us and deploys our app.

Finally, our app will look like this –

You can search the news for your favorite celebrity and get a summarized version of each article.

Try this app I created here Short News App · Streamlit

Conclusion

We successfully built our short news app leveraging AI capabilities. AI has made our life simple. More features can be added to this app. Try it.

Read my other articles here –

• Live Twitter Sentiment Analyzer with Streamlit, Tweepy and Huggingface (analyticsvidhya.com)
• NeatText Library | Pre-processing textual data with NeatText library (analyticsvidhya.com)

Image sources:

Image-1:  https://newspaper.readthedocs.io/en/latest/

Image-2:  https://huggingface.co/