Pagination helps reduce overall loading speed and ensures better crawling and indexing. In this article, we’ll explore the concept of pagination in NewsData.io.
How Pagination works
To paginate in Newsdata.io News API, follow the steps mentioned below:
https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=YOUR_QUERY
This request will retrieve 10 articles/page for free and 50 articles/page for paid users.
To go to the next page, you need to use the nextPage string value given at the bottom of the response. Use that nextPage string value in your query using this parameter: ‘page=Nextpagestring’.
https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=YOUR-QUERY&page=XXXPPPXXXXXXXXXX
This parameter will take you to the next page with a new set of articles.
For instance, my query is Football. The request parameter would be as follows:
https://newsdata.io/api/1/latest?apikey=YOUR_API_KEY&q=football
After entering the query, you will get the data related to your keyword.
Now, go to the bottom of the page, and you will find the “nextPage” with the specified number.
Copy that number and enter it in the query with the page parameter. This request will retrieve the next page.
You need to repeat this process manually to move to the next pages.
Pagination In Python
Mentioned below is the process of Pagination in Python:
- Installing newsdataapi in your Python Library
pip install newsdataapi
- Importing ‘NewsDataApiClient’ package to your program
from newsdataapi import NewsDataApiClient
- Authorizing your API key
api = NewsDataApiClient(apikey="YOUR_API_KEY")
response = api.news_api()
Between the brackets, you can add parameters of your choice, like q, qInTitle, qInMeta, category, etc.
For detailed information, click here.
Methods to Paginate in Python
There are three methods to paginate in Python:
1. Manual Pagination
After searching your query, add the nextPage string value that will then redirect you to the next page.
response = api.news_api(page = "nextPage value")
print (response)
You will have to insert the nextPage string value to the query each time manually.
Note: You can only go to the next page from your current page, not the other way around.
2. Automated Pagination
Users can set the loop and paginate until the last page by inserting this code.
page=None
while True:
response = api.news_api(page=page)
page = response.get('nextPage',None)
if not page:
break
However, once all the API credits are exhausted, data extraction will be stopped.
For instance, if your API credit limits can fetch 200 articles, then it will stop fetching after getting all 200 articles from the response.
3. Scroll
This allows you to gather all the available articles that are there, all at once. It is especially beneficial for users with a high number of API credits.
The users can use the ‘scroll=True‘ parameter to avail of this benefit.
api = NewsDataApiClient(apikey="YOUR_API_KEY")
response = api.news_api(q='pizza',scroll=True)
print(response)
Max Request
Users exhaust their credits while using the scroll parameter. To prevent that from happening, you can use the ‘max_request’ parameter.
api = NewsDataApiClient(apikey="YOUR_API_KEY")
response = api.news_api(q='pizza',scroll=True,max_result=1000)
print(response)
By using this query, Newsdata.io API would fetch only 1000 articles from the total number of articles available.
Furthermore, you can even add ‘debug=True’ to monitor your API credit consumption.
api = NewsDataApiClient(apikey="YOUR_API_KEY",debug=True)
response = api.news_api(q='pizza')
print(response)
To learn more about it, click here.
Important Note
One must note that by using the ‘scroll=True’ query, all the articles available related to your query are fetched which isn’t always an ideal situation.
For Instance,
Your searched query has over 10,000 articles but your API limit is 8,000. Then, the entire 8,000 articles worth of credit would be consumed, and it would show an error stating that all the API credits have been consumed without fetching me any data.
In other words, one who is aware of their API credits should use this parameter. Otherwise, insufficient credits would lead to a failed request.
That is also when the ‘max_result‘ parameter is very useful for those with fewer API credit limits.
Hello, Curious Minds!
Welcome to my corner of the digital world, a space brimming with words and woven with ideas. Fresh out of the rigorous trenches of an Economics honors degree at the esteemed University of Delhi, I know a thing or two about crunching numbers and dissecting trends. But beyond the world of graphs and equations, lies my love for reading and writing. Admittedly, I’m a newbie in the content writing scene, still tasting the ink of fresh beginnings. I believe every corner of life holds a story waiting to be told, and I’m eager to be your storyteller. So, strap yourselves in, dear readers, and let’s dive into the captivating world of words together!
P.S. Feel free to drop a comment or reach out – I’m always up for a good conversation!
