{"id":5210,"date":"2025-01-08T15:59:20","date_gmt":"2025-01-08T10:29:20","guid":{"rendered":"https:\/\/newsdata.io\/blog\/?p=5210"},"modified":"2025-01-08T17:56:36","modified_gmt":"2025-01-08T12:26:36","slug":"streamlit-transforming-newsdata","status":"publish","type":"post","link":"https:\/\/newsdata.io\/blog\/streamlit-transforming-newsdata\/","title":{"rendered":"Streamlit: Transforming News Data Into Actionable Insights"},"content":{"rendered":"[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/4″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid” column_padding_type=”default” gradient_type=”default” offset=”vc_hidden-sm vc_hidden-xs”][\/vc_column][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” el_class=”text_block_wrapper” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”3\/4″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid” column_padding_type=”default” gradient_type=”default” offset=”vc_col-lg-9 vc_col-md-12″][image_with_animation image_url=”5231″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]In today’s fast-paced digital world, staying on top of news trends and sentiment patterns is crucial for businesses and analysts alike. I’m excited to share with you a powerful News Intelligence Dashboard I’ve developed using Python and Streamlit<\/a> that transforms how we process and understand news data. This tool doesn’t just aggregate news\u2014it provides deep insights through sentiment analysis, temporal patterns, and regional distribution of news coverage. By leveraging the NewsData.io<\/a> News API and modern data visualization libraries, we’ll be creating a dynamic dashboard that turns raw news data into actionable intelligence. Whether you’re a data analyst, business strategist, or news enthusiast, this dashboard offers a comprehensive view of news patterns that can inform your decision-making process.[\/vc_column_text][vc_column_text]\n

The Problem and Solution: Making Sense of News Data<\/h2>\n[\/vc_column_text][vc_column_text]In today’s information-rich environment, organizations face a significant challenge: how to effectively monitor and analyze the vast ocean of news content being published every second. Manual news tracking is not just time-consuming\u2014it’s practically impossible to maintain consistency and catch every relevant story. This is where the power of NewsData.io’s API comes into play.<\/p>\n

Consider a scenario where you’re tracking market sentiment about your company or monitoring global events affecting your industry. Without automation, you might spend hours scanning various news sources, trying to piece together patterns and insights. The NewsData.io News API solves this by providing structured access to news from over 82,000 sources across different countries and languages, complete with AI-powered sentiment analysis and regional classification.<\/p>\n

NewsData.io stands beyond simple keyword searches and is at the forefront of this evolution with its sophisticated AI models. While traditional news APIs might give you basic article access, NewsData.io employs state-of-the-art models to provide deeper, more nuanced insights into news content.<\/p>\n

It uses advanced natural language processing to understand the emotional tone and implications of news articles, going beyond simple positive\/negative classifications to capture subtle nuances in reporting. Its Named Entity Recognition (NER) capability automatically identifies and categorizes key entities like companies, and locations, making it invaluable for relationship mapping and trend analysis.<\/p>\n

The dashboard transforms this challenge into an opportunity by automating the entire process. When you input a keyword, the system leverages NewsData.io’s advanced features like sentiment analysis and AI region classification to provide immediate insights. For instance, if you’re tracking cryptocurrency news, the dashboard can instantly show you whether the overall sentiment is positive or negative, which regions are discussing it most actively, and how the coverage has evolved over time.[\/vc_column_text][vc_column_text]\n

Technical Architecture: Building a Robust News Analytics Platform<\/h2>\n[\/vc_column_text][vc_column_text]The dashboard leverages a powerful combination of Python libraries like Pandas<\/a>, Plotly<\/a>, Request, and Streamlit. We will be going basic architecture shown below:[\/vc_column_text][image_with_animation image_url=”5220″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]At the heart of our architecture lies Streamlit, a powerful Python framework that transforms data scripts into shareable web applications. Streamlit handles our frontend rendering and user interactions, making it possible to create an intuitive interface where users can input search parameters, select date ranges, and filter by region (continent, country, city, etc ). The framework’s reactive nature means our dashboard updates in real-time as users modify their search criteria. The image shown below is the basic frame of the dashboard:[\/vc_column_text][image_with_animation image_url=”5216″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]The data is gathered using NewsData.io API with the help of the following code.The integration is handled through a dedicated NewsAnalyzer class, which manages API requests, pagination, and error handling. It has robust error checking and rate limiting to ensure reliable data retrieval. The API provides rich metadata including sentiment analysis and regional classification, which we leverage for our analytics.<\/p>\n

You can see in the code that I\u2019m using the \u201carchive<\/strong>\u201d endpoint. I can use this endpoint because I have a \u201cCorporate<\/strong>\u201d plan. By using this plan I can fetch last two years data from the present date. Additionally, I can customize this plan if I need more historical news. NewsData.io can provide the news from 2021 till present.[\/vc_column_text][vc_column_text]\n

class NewsAnalyzer:\n    def __init__(self, api_key):\n        self.api_key = api_key\n    def fetch_and_analyze_news(self, keyword, ai_region, from_date=None, to_date=None):\n        base_url = "https:\/\/newsdata.io\/api\/1\/archive"\n        params = {\n            "apikey": self.api_key,\n            "q": keyword,\n        }\n        if from_date:\n            params["from_date"] = from_date\n        if to_date:\n            params["to_date"] = to_date\n        all_results = []\n        try:\n            response = requests.get(base_url, params=params)\n            if response.status_code == 200:\n                data = response.json()\n                # Process and filter results\n                for article in data.get('results', []):\n                    ai_regions = article.get('ai_region', [])\n                    if isinstance(ai_regions, str):\n                        ai_regions = [ai_regions]\n                    article_regions = [r.lower() for r in ai_regions if r]\n                    if not ai_region or any(ai_region.lower() in r for r in article_regions):\n                        all_results.append(article)\n                # Handle pagination\n                while data.get('nextPage'):\n                    params['page'] = data['nextPage']\n                    response = requests.get(base_url, params=params)\n                    if response.status_code == 200:\n                        data = response.json()\n                        for article in data.get('results', []):\n                            ai_regions = article.get('ai_region', [])\n                            if isinstance(ai_regions, str):\n                                ai_regions = [ai_regions]\n                            article_regions = [r.lower() for r in ai_regions if r]\n                            if not ai_region or any(ai_region.lower() in r for r in article_regions):\n                                all_results.append(article)\n                    else:\n                        break\n            return all_results\n        except Exception as e:\n            print(f"Error fetching news: {str(e)}")\n            return []\n<\/code><\/pre>\n[\/vc_column_text][vc_column_text]The data processing layer is built on Pandas, Python’s data manipulation library. When news data flows in from the NewsData.io API, Pandas efficiently structures it into DataFrames, enabling fast aggregations and transformations. The code demonstrates this in the process_news_data function:<\/p>\n
Similar read: Textual data preprocessing<\/a><\/em><\/strong>[\/vc_column_text][vc_column_text]\n
def process_news_data(news_data):\n    processed_data = []\n    for article in news_data:\n        # Transform raw API data into structured format\n        record = {\n            'title': article.get('title', ''),\n            'sentiment': article.get('sentiment', 'neutral'),\n            'ai_region': article.get('ai_region', []),\n            'pubDate': pd.to_datetime(article.get('pubDate', ''))\n        }\n        processed_data.append(record)\n    \n    df = pd.DataFrame(processed_data)\n    df['date'] = df['pubDate'].dt.date\n    df['hour'] = df['pubDate'].dt.hour\n    return df\n<\/code><\/pre>\n[\/vc_column_text][vc_column_text]The NewsData.io API integration is handled through a dedicated NewsAnalyzer class, which manages API requests, pagination, and error handling. It has robust error checking and rate limiting to ensure reliable data retrieval. The API provides rich metadata including sentiment analysis and regional classification, which we leverage for our analytics.<\/p>\n
For visualization, we use Plotly Express, which creates interactive charts that users can zoom, pan, and hover over for detailed information. The visualization layer transforms our processed DataFrame into insightful charts showing sentiment distribution, temporal patterns, and source analysis.[\/vc_column_text][vc_column_text]Our data processing pipeline follows a clear flow:<\/p>\n
    \n
  1. User input validation and parameter processing<\/li>\n
  2. Batched API requests with progress tracking<\/li>\n
  3. Data cleaning and structuring using Pandas<\/li>\n
  4. Feature extraction (temporal, regional, sentiment)<\/li>\n
  5. Interactive visualization generation<\/li>\n<\/ol>\n
    This architecture ensures scalability while maintaining responsive performance, even when processing thousands of news articles. The modular design also makes it easy to extend functionality or add new visualization types as needed.[\/vc_column_text][vc_column_text]\n
    Key Features Deep Dive: Making the Most of Your News Intelligence Dashboard<\/h2>\n[\/vc_column_text][vc_column_text]The \u201cq<\/strong>\u201d parameter in the URL uses elastic search<\/a> to fetch the data.\u00a0 When you enter a keyword in the search bar, it’s automatically encoded into the URL query parameters, making your searches shareable and bookmarkable. For example, if you search for “artificial intelligence”, the URL updates to something like:<\/p>\n
    https:\/\/newsdata.io\/api\/1\/archive?apikey=YOUR_API_KEY&q=artificial%20intelligence<\/em><\/strong>[\/vc_column_text][vc_column_text]Search Functionality and Parameter Customization: <\/strong>The dashboard’s search interface is designed for precision and flexibility. Users can search by keywords and filter results using a carefully curated list of regions. The date range selector enforces a two-year maximum span to ensure optimal performance.<\/p>\n
    At the core of our dashboard is an intelligent search system that leverages NewsData.io’s elastic search capabilities. When you enter a keyword in the search bar, it’s automatically encoded into the URL query parameters, making your searches shareable and bookmarkable.<\/p>\n
    Here’s how our search parameters work:[\/vc_column_text][vc_column_text]\n
    keyword = st.sidebar.text_input("Enter keyword (required)") \nregion = st.sidebar.selectbox("Select region (optional)", countries) \ndate_range = st.sidebar.date_input(      "Select date range (max 2 years)", \n[min_date, max_date], min_value=min_date, max_value=max_date )<\/code><\/pre>\n[\/vc_column_text][image_with_animation image_url=”5215″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]Sentiment Analysis Visualization: <\/strong>The sentiment analysis feature processes NewsData.io’s pre-classified sentiment data (positive, negative, neutral) and presents it through an interactive pie chart. This gives you immediate insight into the emotional tone of news coverage:[\/vc_column_text][vc_column_text]\n
    keyword = st.sidebar.text_input("Enter keyword (required)") \nregion = st.sidebar.selectbox("Select region (optional)", countries) \ndate_range = st.sidebar.date_input(      "Select date range (max 2 years)", \n[min_date, max_date], min_value=min_date, max_value=max_date )<\/code><\/pre>\n[\/vc_column_text][image_with_animation image_url=”5224″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]Temporal Pattern Analysis: <\/strong>The dashboard includes a sophisticated temporal analysis system that breaks down news coverage by hour of publication. This helps identify peak news times and coverage patterns:[\/vc_column_text][vc_column_text]\n
    hourly_dist = df['hour'].value_counts().sort_index()\n\nfig_time = px.line( x=hourly_dist.index, y=hourly_dist.values, title='Publication Time Distribution', labels={'x': 'Hour of Day', 'y': 'Number of Articles'} )<\/code><\/pre>\n[\/vc_column_text][image_with_animation image_url=”5225″ image_size=”full” animation_type=”entrance” animation=”None” animation_movement_type=”transform_y” hover_animation=”none” alignment=”” border_radius=”none” box_shadow=”none” image_loading=”default” max_width=”100%” max_width_mobile=”default”][vc_column_text]\n
    Practical Applications: Turning News Data into Business Intelligence<\/h3>\n[\/vc_column_text][vc_column_text]Understanding how to apply the News Intelligence Dashboard across different industries can transform how organizations monitor and respond to news coverage. Let me walk you through several real-world applications and show you how to extract meaningful insights from the visualizations.<\/p>\n
    Financial Services Industry: <\/strong>Imagine you’re a financial analyst tracking market sentiment around cryptocurrency. By entering “Bitcoin” as your keyword, the dashboard provides immediate insights through its visualization suite. The sentiment distribution chart might show 60% neutral, 25% positive, and 15% negative coverage. This pattern, especially when viewed across time, can correlate with market movements and help inform trading strategies. The temporal analysis becomes particularly valuable during market hours, showing you how news sentiment shifts during key trading periods.<\/p>\n
    Healthcare and Pharmaceutical Companies:<\/strong> For healthcare organizations, monitoring public health trends and pharmaceutical developments is crucial. Let’s say you’re tracking coverage of a new vaccine. The regional distribution insights become invaluable here, showing which areas are discussing the topic most actively. The dashboard’s temporal analysis might reveal that scientific announcements tend to generate coverage waves that peak after 48-72 hours, helping you plan your communication strategy.<\/p>\n
    Technology Companies:<\/strong> Tech companies can use the dashboard to monitor competitive intelligence and industry trends. When tracking topics like “artificial intelligence” or “cloud computing,” the source analysis feature helps identify which publications are driving the conversation. The real-time processing capabilities mean you can spot emerging stories and respond quickly to market developments.<\/p>\n
    Interpreting the Visualizations:<\/strong><\/p>\n
    1. Sentiment Analysis Pie Chart:<\/p>\n