All About Pagination in NewsData.io News API

Pagination helps reduce overall loading speed and ensures better crawling and indexing. In this article, we’ll explore the concept of pagination in NewsData.io.

How Pagination works

To paginate in Newsdata.io News API, follow the steps mentioned below:

https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=YOUR_QUERY

This request will retrieve 10 articles/page for free and 50 articles/page for paid users.

To go to the next page, you need to use the nextPage string value given at the bottom of the response. Use that nextPage string value in your query using this parameter: ‘page=Nextpagestring’.

https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=YOUR-QUERY&page=XXXPPPXXXXXXXXXX

This parameter will take you to the next page with a new set of articles.

For instance, my query is Football. The request parameter would be as follows:

https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=football

After entering the query, you will get the data related to your keyword.

Now, go to the bottom of the page, and you will find the “nextPage” with the specified number.

Copy that number and enter it in the query with the page parameter. This request will retrieve the next page.

You need to repeat this process manually to move to the next pages.

Pagination In Python

Mentioned below is the process of Pagination in Python:

• Installing newsdataapi in your Python Library

pip install newsdataapi

• Importing ‘NewsDataApiClient’ package to your program

from newsdataapi import NewsDataApiClient

• Authorizing your API key

api = NewsDataApiClient(apikey="YOUR_API_KEY")

response = api.news_api()

Between the brackets, you can add parameters of your choice, like q, qInTitle, qInMeta, category, etc.

For detailed information, click here.

Methods to Paginate in Python

There are three methods to paginate in Python:

1. Manual Pagination

After searching your query, add the nextPage string value that will then redirect you to the next page.

response = api.news_api(page = "nextPage value")

print (response)

You will have to insert the nextPage string value to the query each time manually.

Note: You can only go to the next page from your current page, not the other way around.

2. Automated Pagination

Users can set the loop and paginate until the last page by inserting this code.

page=None

while True:

response = api.news_api(page=page)

page = response.get('nextPage',None)

if not page:

break

However, once all the API credits are exhausted, data extraction will be stopped.

For instance, if your API credit limits can fetch 200 articles, then it will stop fetching after getting all 200 articles from the response.

3. Scroll

This allows you to gather all the available articles that are there, all at once. It is especially beneficial for users with a high number of API credits.

The users can use the ‘scroll=True‘ parameter to avail of this benefit.

api = NewsDataApiClient(apikey="YOUR_API_KEY")

response = api.news_api(q='pizza',scroll=True)

print(response)

Max Request

Users exhaust their credits while using the scroll parameter. To prevent that from happening, you can use the ‘max_request’ parameter.

api = NewsDataApiClient(apikey="YOUR_API_KEY")

response = api.news_api(q='pizza',scroll=True,max_result=1000)

print(response)

By using this query, Newsdata.io API would fetch only 1000 articles from the total number of articles available.

Furthermore, you can even add ‘debug=True’ to monitor your API credit consumption.

api = NewsDataApiClient(apikey="YOUR_API_KEY",debug=True)

response = api.news_api(q='pizza')

print(response)

To learn more about it, click here.

Important Note

One must note that by using the ‘scroll=True’ query, all the articles available related to your query are fetched which isn’t always an ideal situation.

For Instance,
Your searched query has over 10,000 articles but your API limit is 8,000. Then, the entire 8,000 articles worth of credit would be consumed, and it would show an error stating that all the API credits have been consumed without fetching me any data.

In other words, one who is aware of their API credits should use this parameter. Otherwise, insufficient credits would lead to a failed request.

That is also when the ‘max_result‘ parameter is very useful for those with fewer API credit limits.