REST API Design: Filtering, Sorting, and Pagination
RESTful APIs have become a cornerstone of modern web development, allowing developers to create powerful, scalable, and efficient web applications. When designing REST APIs, it's essential to consider key features like filtering, sorting, and pagination to ensure optimal user experience and server performance.
Filtering allows users to narrow down search results by defining specific criteria, while sorting allows results to be ordered in a particular manner. Pagination enables the API to return a subset of data, reducing the amount of data transferred and improving performance.
In this blog, we will discuss the importance of filtering, sorting, and pagination in REST API design. We will also explore the different pagination techniques used in APIs, such as cursor-based and offset-based pagination, and provide examples of their implementation.
Additionally, we'll cover best practices for filtering and sorting data in API requests, including query parameter syntax and sorting algorithms. By the end of this blog, you'll have a solid understanding of how to design and implement a RESTful API with filtering, sorting, and pagination.
Table of Content
Filtering
Filtering in REST API refers to the process of limiting the result set of an API request based on specific criteria.
Filtering allows clients of a REST API to retrieve only the data they need, and can be used to improve the performance of the API and reduce the amount of data transmitted over the network.
For example, a request to retrieve a list of items with a certain price range might look like this:
GET /items?price=20-60In this example, the query parameter price=20-60 limits the response to items with a price between 20 and 60. This allows the client to retrieve only the data it needs, reducing the amount of data transferred and improving performance.
Filtering Methods
There are several ways to implement filtering in a REST API:
1. Path Parameters
In a REST API, path parameters can be used to filter data by encoding the filtering conditions as part of the URL path.
For example, consider an API endpoint that returns a list of products. You can use path parameters to filter the products based on specific criteria. Let's say you want to get all the products that belong to a specific category. You can use a path parameter like {category} to filter the products by category.
The API endpoint would look like this:
GET /products/category/{category}If you want to get all the products in the "Electronics" category, you can make a request like this:
GET /products/category/ElectronicsThe API would then return a list of all the products that belong to the "Electronics" category.
In this example, the path parameter {category} is used to filter the products by category. The value of the path parameter is extracted from the URL and used to retrieve the relevant data.
This method of filtering is particularly useful when you have a limited set of predefined filtering conditions, such as author name or genre, that can be easily encoded as part of the URL path.
2. Query Parameters
It filters data based on specific query parameter values. These parameters are added to the URL after a ? and separated by &. The value of the query parameter is used to filter data.
For example, consider an API endpoint that returns a list of products. You can use query parameters to filter the products based on specific criteria. Let's say you want to get all the products that have a price greater than a certain amount. You can use a query parameter like price_gt to filter the products by price.
The API endpoint would look like this:
GET /products?price_gt=50If you want to get all the products with a price greater than $50, you can make a request like this:
GET /products?price_gt=50The API would then return a list of all the products with a price greater than $50.
In this example, the query parameter price_gt is used to filter the products by price. The value of the query parameter is extracted from the URL and used to retrieve the relevant data.
This method of filtering is the most common and flexible method for filtering in REST APIs, as it allows you to specify multiple filtering conditions in a simple and intuitive manner.
Query parameters can be easily added to the URL, and can be combined in various ways to filter data based on complex conditions.
3. Request Body
Occasionally, you may need to pass complex filtering conditions to an API that cannot be expressed as simple query parameters. For these cases, you can send the filtering conditions as a JSON or XML payload in the request body.
Here's an example of using the request body to filter a list of books based on multiple conditions, such as author name, publication year, and minimum rating:
POST /books/filter
Content-Type: application/json
{
"author": "Jane Austen",
"year": 1813,
"rating": 4
}In this example, the API endpoint /books/filter is expecting a JSON payload in the request body that defines the filtering conditions. The payload includes the author name, publication year, and minimum rating, and the API will return a filtered list of books that meet all of these conditions.
This method of filtering is particularly useful when you need to pass complex or multiple filtering conditions to the API that cannot be easily represented using query parameters or path parameters.
Comparison and conjunction operators in filtering
Comparison and conjunction operators are used in filtering in REST API to specify the conditions that must be met for a resource to be included in the response.
Comparison operators are used to compare values and include operators such as:
- Equal to (=)
- Not equal to (!=)
- Less than (<)
- Less than or equal to (<=)
- Greater than (>)
- Greater than or equal to (>=)
Conjunction operators allow you to combine multiple conditions and determine whether they are all true or not. Common conjunction operators include:
&(and)|(or)
Example 1
GET /products?price=>10&price=<50In this example, the > and < comparison operators are used to specify that only products with a price greater than 10 and less than 50 should be returned in the response.
Example 2
For example, in a REST API that returns a list of books, you could filter the results to only include books with a specific author and a publication year greater than a certain value:
GET /books?author=Jane_Austen&year>=1800This would return a list of books written by Jane Austen and published after 1800.
Conjunction operators are used to combine multiple conditions, such as AND (&) and OR (|). By combining comparison and conjunction operators, clients can create complex filtering conditions to retrieve exactly the data they need. The specific operators and syntax used may vary depending on the API implementation.
Sorting
Sorting in REST API refers to the ability to arrange the data returned by a REST API endpoint in a specific order, either in ascending or descending order, based on one or more fields.
This allows clients to request the data in a desired order, making it easier to find what they are looking for. The specific sorting mechanism used by a REST API will depend on the API's design and implementation.
1. Ascending sorting
Ascending sorting is a technique used in REST API to arrange data in an ascending order based on a specific field. The ascending order is arranged in such a way that the lowest value of the field is displayed first, and the highest value of the field is displayed last. This method is used to filter data in REST API.
For example, consider an API endpoint that returns a list of products. You can use ascending sorting to sort the products based on their price. You can use a query parameter like sort=price&order=asc to sort the products by their price in ascending order.
The API endpoint would look like this:
GET /products?sort=price&order=ascThe API endpoint then retrieves a list of all the products and sorts them in ascending order based on their price field. The resulting list will display products with the lowest price at the top of the list and products with the highest price at the bottom of the list.
For example, if the API returns the following products:
Product 1: Name: Laptop, Price: $500, Rating: 4.5
Product 2: Name: Smartphone, Price: $600, Rating: 4.2
Product 3: Name: Headphones, Price: $50, Rating: 3.9
Product 4: Name: Smartwatch, Price: $200, Rating: 4.1After applying ascending sorting based on the price field, the resulting list will be:
Product 3: Name: Headphones, Price: $50, Rating: 3.9
Product 4: Name: Smartwatch, Price: $200, Rating: 4.1
Product 1: Name: Laptop, Price: $500, Rating: 4.5
Product 2: Name: Smartphone, Price: $600, Rating: 4.2In this example, the sort=price&order=asc query parameter is used to sort the products in ascending order based on their price field. The resulting list is sorted from the lowest price to the highest price, making it easier for the client to find the products they're looking for.
2. Descending sorting
Descending sorting is a type of data sorting method used in REST APIs to arrange data based on a specific field in descending order. This means that the data is arranged from the highest value of the field to the lowest value of the field.
Here's an example to explain descending sorting in REST API:
Consider a music streaming application that has a REST API to retrieve a list of songs. Each song has a title, artist, and length in seconds. To retrieve a list of songs sorted by length in descending order, the client can make a GET request to the API endpoint with a query parameter of sort=length and order=desc.
GET /songs?sort=length&order=descThe API endpoint then retrieves a list of all the songs and sorts them in descending order based on their length field. The resulting list will display songs with the highest length at the top of the list and songs with the lowest length at the bottom of the list.
3. Multiple column sorting
Multiple column sorting is a filtering technique used in REST APIs to sort data based on multiple columns. This technique is useful when sorting data based on a single column does not provide enough information to make informed decisions.
Here's an example to explain multiple column sorting in REST APIs:
Consider an e-commerce application that has a REST API to retrieve a list of products. Each product has a name, price, category, and popularity score. To retrieve a list of products sorted by their popularity score in descending order and then by their price in ascending order, the client can make a GET request to the API endpoint with query parameters of sort=-popularity,price.
GET /products?sort=-popularity,priceThe API endpoint then retrieves a list of all the products and sorts them first by the popularity score field in descending order and then by the price field in ascending order.
The resulting list will display products with the highest popularity score at the top of the list and products with the same popularity score will be sorted by their price in ascending order.
For example, if the API returns the following products:
Product 1: Name: T-shirt, Price: $10, Category: Clothing, Popularity Score: 8
Product 2: Name: Headphones, Price: $50, Category: Electronics, Popularity Score: 9
Product 3: Name: Backpack, Price: $30, Category: Accessories, Popularity Score: 8
Product 4: Name: Smartphone, Price: $500, Category: Electronics, Popularity Score: 10After applying multiple column sorting based on the popularity score field in descending order and the price field in ascending order, the resulting list will be:
Product 4: Name: Smartphone, Price: $500, Category: Electronics, Popularity Score: 10
Product 2: Name: Headphones, Price: $50, Category: Electronics, Popularity Score: 9
Product 1: Name: T-shirt, Price: $10, Category: Clothing, Popularity Score: 8
Product 3: Name: Backpack, Price: $30, Category: Accessories, Popularity Score: 8Pagination
Pagination in REST API refers to the process of dividing a large number of results into smaller parts (known as pages) and sending them to the client in separate requests. This helps to manage the size of response data and reduce the amount of data that needs to be transferred over the network.
The client can request a specific page by including parameters in the API request that specify the desired page number and the number of items per page.
The API will then respond with the requested page of results along with metadata about the total number of results and the number of pages available. This approach helps to reduce network bandwidth usage and improve the speed and efficiency of API requests.
1. Offset Pagination
Offset pagination is a widely used method in RESTful APIs that facilitates the return of a selected subset of data from a larger collection. With this technique, developers can specify an offset value and a limit value, which enable them to determine which specific portion of results is to be returned. Offset pagination is usually paired with sorting to make it easier for users to paginate through voluminous data sets.
Let's say we have a RESTful API that provides a list of products. To implement offset pagination, we can use the GET method with the following URL:
https://api.example.com/products?limit=10&offset=20In this example, we're requesting a list of 10 products starting from the 21st product (since offset is 20-based). We're using the limit parameter to specify the maximum number of items we want to return, and the offset parameter to specify the starting point of the result set.
The API server can return a JSON response that looks like this:
{
"data": [
{
"id": 21,
"name": "Product 21",
"description": "Description of product 21",
"price": 10.99
},
{
"id": 22,
"name": "Product 22",
"description": "Description of product 22",
"price": 15.99
},
...
],
"metadata": {
"total_count": 100,
"limit": 10,
"offset": 20
}
}In this response, the data field contains the list of 10 products starting from the 21st product. The metadata field provides information about the total number of products available (total_count), the limit parameter we used to request the data, and the offset value we specified.
By using offset pagination, we can efficiently retrieve large sets of data while minimizing the amount of data transferred and improving server performance. However, it's essential to keep in mind that offset pagination can have issues with consistency and accuracy when dealing with frequently updated data. In such cases, using cursor-based pagination is recommended.
2. Cursor-based pagination
Cursor-based pagination is a common technique used in RESTful APIs for paginating through large sets of data. In this technique, a cursor is used as a pointer to a specific location in the data set, and the API client retrieves the next page of results using the cursor.
Here's an example of how cursor-based pagination works in a RESTful API:
Let's assume that we have a database of user profiles with millions of records, and we want to provide a paginated API to return these user profiles.
Step-1: The API client makes an initial request to the server, specifying the number of items to be returned per page and a starting cursor value.
GET /users?limit=50&cursor=0
Step-2: The server returns the first 50 user profiles, along with a cursor value to use for the next page.
{
"data": [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"},
{"id": 3, "name": "Charlie"},
...
{"id": 50, "name": "Zoe"}
],
"cursor": "eyJpZCI6MzQ2NTAsInNlcXVlbmNlIjozNTYyMH0="
}Step-3: The API client makes a subsequent request to retrieve the next page of data, using the cursor value returned by the server.
GET /users?limit=50&cursor=eyJpZCI6MzQ2NTAsInNlcXVlbmNlIjozNTYyMH0=Step-4: The server retrieves the next 50 user profiles from the database, starting from the position indicated by the cursor value. It then returns the next page of results, along with a new cursor value to use for the next page.
{
"data": [
{"id": 51, "name": "Alex"},
{"id": 52, "name": "Ben"},
{"id": 53, "name": "David"},
...
{"id": 100, "name": "Oliver"}
],
"cursor": "eyJpZCI6MTAwMDEsInNlcXVlbmNlIjozNTYyMX0="
}Step-5: The process repeats until all data has been retrieved.
Cursor-based pagination is a flexible and efficient way to paginate through large sets of data, and it can be customized to handle different sorting and filtering requirements.
3. Keyset pagination
Keyset pagination is used to retrieve the next set of results in a sorted data set. To use keyset pagination, the results must be sorted by one or more columns, and each item in the result set must have a unique identifier that can be used to determine the next set of results.
For instance, if we have a database table with columns such as id, name, and date_created, we could use the date_created column to sort the data.
To implement keyset pagination, we need to choose one or more columns to sort the data. Let's say we choose to sort by the date_created column.
First, we'll fetch the first set of results by running the following query:
SELECT id, name, date_created FROM my_table ORDER BY date_created LIMIT 6;This will return the first 10 results, sorted by date_created. We'll also retrieve the last item's value for the sorting column, in this case, date_created.
Now, to fetch the next set of results, we'll run the following query:
SELECT id, name, date_created FROM my_table WHERE date_created > '2022-01-07' ORDER BY date_created LIMIT 6;This query uses the last date_created value from the previous query to get the next set of results. It fetches all the rows whose date_created value is greater than the last date_created value and returns the next 10 results.
By using the unique identifier (`date_created` in this example) instead of an offset value, keyset pagination provides better performance and avoids the issues of duplicate data and inconsistent results that can occur with offset pagination.
4. Seek Pagination
Seek-based pagination is used to retrieve a subset of results in a sorted data set. It works by specifying a starting point and returning results that come after that starting point, up to a certain limit. Seek-based pagination is often used when dealing with large datasets and can be faster than other types of pagination.
For example, let's say we have a database table of user accounts with columns such as id, username, and registration_date. We want to retrieve the first 50 users who registered after a specific date. To use seek-based pagination, we would first sort the data by registration_date in ascending order. We would then use the registration_date of the first user who registered after our specified date as the starting point and retrieve the next 50 users from that point.
Let's say our specified date is January 1, 2022, and the first user who registered after that date has a registration_date of January 5, 2022. We would use the registration_date of this user as the starting point and retrieve the next 50 users from that point. If we want to retrieve the next 50 users after that, we would use the registration_date of the 50th user as the new starting point and continue the process.
API Pagination implementation - Twitter
Twitter's API uses cursor-based pagination to return results in a set number of tweets per request, or "page". The API will return a next_cursor value in the response, which can be used as the "cursor" parameter in the next API request to retrieve the next set of tweets.
This process can be repeated until the next_cursor value is 0, indicating that there are no more pages of results to retrieve. In cursor-based pagination, the API returns a cursor that indicates the position of the next set of results, allowing you to retrieve subsequent pages of results. The API also returns a previous cursor that you can use to retrieve previous pages of results.
Twitter also provides a max_id parameter, which you can use to retrieve results with an ID less than or equal to the specified value. This can be useful for pagination when working with a large number of results.
API Pagination implementation - Spotify
Spotify's API uses offset-based pagination to return results. This means that each API request returns a set number of results, starting at a specified offset index.
This process can be repeated until there are no more results to retrieve. The Spotify API will return the total number of items in the response, which can be used to determine when all the results have been retrieved.
Best Practices for Filtering, Sorting and Pagination
- Use standard query parameters: Use standard query parameters for filtering, sorting, and pagination, such as "filter", "sort", "page", and "limit". This makes it easier for developers to understand and use your API.
- Limit the number of resources returned: Avoid returning large data sets in a single API request. Instead, limit the resources returned and provide pagination parameters to allow clients to retrieve additional data.
- Use efficient sorting algorithms: Use efficient sorting algorithms that can handle large data sets, such as quicksort or merge sort. Avoid using bubble or insertion sort, which are less efficient for large data sets.
- Use index fields for filtering: Use index fields for filtering data, as it can significantly improve query performance. Make sure to index the fields that are most commonly used for filtering.
- Allow for multiple filter parameters: Allow for multiple filter parameters to be used in a single API request, and provide logical operators like "AND" and "OR" to allow for complex filtering.
- Use consistent sorting criteria: Use consistent sorting criteria to ensure that the same data is returned in the same order for each API request. Please allow clients to specify the sorting criteria.
- Provide default sorting: Provide default sorting criteria if clients don't specify any sorting parameters. This can improve the user experience by returning results in a predictable order.
- Use caching: Use caching to improve API performance and reduce the load on the server. Cache frequently requested data and invalidate the cache when data changes.
Conclusion
RESTful APIs follow a stateless, client-server model, where the client makes requests to the server to access or manipulate data. REST APIs are scalable, flexible, and can be easily utilized by various systems, including browsers, mobile devices, and servers. They are also typically easier to develop and test compared to other types of APIs.
By incorporating filtering, sorting, and pagination into API design, developers can create more efficient and user-friendly applications that meet the needs of their users.
Whether you are building a small-scale application or a large-scale enterprise system, adopting these design principles will help you create a robust and reliable REST API that meets the needs of your users.
Atatus API Monitoring and Observability
Atatus provides Powerful API Observability to help you debug and prevent API issues. It monitors the consumer experience and is notified when abnormalities or issues arise. You can deeply understand who is using your APIs, how they are used, and the payloads they are sending.
Atatus's user-centric API observability tracks how your actual customers experience your APIs and applications. Customers may easily get metrics on their quota usage, SLAs, and more.
It monitors the functionality, availability, and performance data of your internal, external, and third-party APIs to see how your actual users interact with the API in your application. It also validates rest APIs and keeps track of metrics like latency, response time, and other performance indicators to ensure your application runs smoothly.