All About Pagination in NewsData.io News API

Pagination, a common technique in APIs, plays a crucial role in breaking down large data sets into manageable portions. In this article, we’ll explore the concept of pagination in the context of the NewsData.io API, understanding its significance and how it enhances data access experience.

Importance of Pagination

Pagination is a method used in APIs to divide large data sets into smaller, more manageable chunks or pages. Rather than bury  the user or the system with an excessive amount of data in a single response, pagination allows for controlled data retrieval, ensuring smoother performance and better user experiences.

How to Utilize Pagination

The pagination method is quite useful for users and easy to use just follow the steps below to understand the pagination of NewsData.io.

For example:
https://newsdata.io/api/1/news?apikey=YOUR_API_KEY&q=YOUR_QUERY

This request will retrieve 10 articles per page if you are a free user and 50 articles per page for paid plans. To go to the next page, you need to use the next page string given at the bottom of the response. Now copy that and paste the page string in your query using this parameter- ‘Page=Nextpagestring’ as shown in the below example.
https://newsdata.io/api/1/news?apikey=YOUR_API_KEY&q=YOUR-QUERY&page=XXXPPPXXXXXXXXXX

Then it will take you to the next page where you will have 50 articles if you are a paid user. Let’s understand how pagination works with an example

My query is Football so the request parameter would be

https://newsdata.io/api/1/news?apikey=YOUR_API_KEY&q=football

Then enter this query and you will get the data related to your keyword then go to the bottom of the page and you will find the next page with the specified number.

Copy that number and enter in the query with &page parameter as shown below. This request will retrieve the next page of the request.

You need to repeat this process to move to the third, fourth, and so on pages and this query will work until the last 50 articles for paid and 10 for free users are left.

Pagination In Python

The process of pagination in Python is quite similar to the steps mentioned above. So let’s understand how users can utilize pagination in Python. Start by installing the “newsdataapi” library using this command

Installing newsdataapi

pip install newsdataapi

This command will install newsdataapi to your python.

From newsdataapi import NewsDataApiClient

This command will authorize your API key, Initialize the client with your API key and then you are ready to make your first request.
api = NewsDataApiClient(apikey="YOUR_API_KEY") response = api.news_api()

Between the brackets, you can add your parameter for example q, category, timezone, etc.

To know more about how to integrate Python client you can visit this blog.

Methods to Paginate in Python

There are basically three ways that users can paginate in Python

• Adding page string: This method requires adding a page string number to the query which is present at the bottom of the response.
• Automatic Pagination: This method will automatically paginate to the next page without manually adding the page string.
• Scroll: This method will fetch all the data that is there to fetch.

First, you need to make your request after that there are three ways by which you can paginate.

1. Adding Page String

After searching your query add the page string that is available at the bottom of the response and you will be redirected to the next page.

response = api.news_api(page = "nextPage value") print (response)

2. Automatic Pagination

Users can set the loop and paginate till the last page without adding the next page value every time manually by setting this parameter.

So to set the automatic article extraction the parameter would be

page=None
while True:
response = api.news_api(page = page)
page = response.get('nextPage',None)
if not page:
break

The data will stop getting fetch responses once your API credits are totally consumed. For example, if your API credit limits can fetch 200 articles then it will stop fetching after getting all 200 articles from response.

3. Scroll

Scroll allows users to gather all the available articles all at once without manually adding page strings and is also beneficial for users with a high number of API credits.

Pagination in Python allows users to extract news articles without adding the page number. By adding the ‘scroll=true‘ parameter user can get the maximum number of data that is there to be presented.

api = NewsDataApiClient(apikey="YOUR_API_KEY")

response = api.news_api(q='pizza',scroll=True)
print(response)

If you want to set the number of articles limit so you don’t end up using all credits during usage of the Scroll parameter. You can use ‘max_request’ parameter By this parameter, users can utilize their credits optimally without failed requests and get the desired number of articles per request.

api = NewsDataApiClient(apikey="YOUR_API_KEY")

response = api.news_api(q='pizza',scroll=True,max_result=1000)
print(response)

By using this query it would fetch only 1000 articles from the total number of articles available.

Note: That you can only go to the next page from your current page.

Max Request

If you dont have sufficient API credits to fetch the data then you can add the max_result parameter and add the number of articles you want. Which will allow users to get only a specific number of articles mentioned.
response = api.news_api(q='pizza',scroll=True,max_result=1000)
print(response)
This query would fetch only 1000 articles from the maximum number of articles. You can even add debug=True to monitor your API credits consumption.

You can visit this site to learn more about the Python client parameters.

Note: It is important to note that by using this query it would fetch all the API credits that your query requires. For instance, if my searched query has over 10,000 articles but my API limit is for 8,000 articles then my entire 8,000 articles worth of credit would be consumed and it would show an error stating that all of your API credits have been consumed without fetching me any data.

So you should only use this parameter when you are not aware of your API credits and it would lead to a failed request if the request exceeds your credit limit. So per API credit would fetch 50 articles for paid and 10 articles for free users. Otherwise, you can use the max_result parameter which is very useful for those users who have fewer API credit limits.

Conclusion

Pagination in the NewsData.io API showcases the platform’s commitment to delivering data efficiently and enhancing user experiences. By dividing extensive news data into manageable pages, the API empowers users to access, explore, and utilize information with ease.

Pagination further contributes to the overall accessibility and usability of the NewsData.io API, making it an ideal choice for applications that require seamless interaction with large data sets.