Rate Limiting (Intentional Slow-Downs) in Some Third-Party Tools

On 20 July 2020 at about 20:30 UTC, Habitica made a change that may cause slowness in some third-party tools (extensions, addons, scripts, etc). Technical details for the creators of the tools are given at the bottom of this post but the main reason for this post is to inform the Habiticans who use those tools.

In brief, Habitica introduced "rate limiting", which means that third-party tools cannot communicate with Habitica too often. Previously, there was no limit on how often a tool could fetch data about your account, or how often it could modify your account (e.g., to cast skills), and because of that some player's private scripts had been doing so many updates so quickly that sometimes Habitica's entire server slowed down and that caused slowness and errors for all players using Habitica. Rate limiting will help to prevent that.

Now that rate limiting has started, if you use third-party tools, scripts, or extensions, then those tools will be able to make at most 30 "requests" each minute on your behalf. A "request" is either fetching information from Habitica (e.g., your Inventory, your tasks, all posts from a Guild, etc) or updating something in Habitica (e.g., adding a task, sending a message, casting a Skill, etc). If a third-party tool makes more than 30 requests in a single minute, then the tool won't be able to make any more requests until the minute is over.

For many tools, this won't have any effect because they don't need to make that many requests that often. However some tools may now be noticeably slower, for example:

If you own a Challenge and are using a tool to fetch the progress of all Challenge participants, then if there are more than 30 participants it will take the tool more than a minute to fetch all of them. If there are hundreds of participants, it make take many minutes before the tool finishes fetching all the data.
If you own a Guild and are using a tool to fetch all the Guild members (e.g., if you want to remove inactive members), then it's likely to take many minutes before the full member list is available. For guilds with several thousand members, it may take a couple of hours before all members are fetched.
If you own a Guild and are using a tool to remove more than 30 Guild members at once, it will take more than a minute.
If you are using a tool to hatch or feed many Pets at once, it may take many minutes.
If you are using a tool to cast Skills in bulk, you may find that it takes longer for all the Skills to be cast.
If you are using a browser extension to display chat posts from many Guilds at once, you may find that not all Guilds can be updated as rapidly as you are used to.

These are just examples of unavoidable slowness that you may experience; other activities not listed here may also be affected.

It's important to note that this affects only third-party tools. Habitica's official website and mobile apps won't be slowed down. It also will not affect any third-party tools that only make cosmetic adjustments to Habitica's website - for example, if you use a Stylish theme to change the website's colours, you won't notice any difference.

The rate limiting is based on requests performed by your own account rather than on your use of a specific tool. This means that if you are using two third-party tools at once, then both tools together can make only 30 requests per minute on your behalf. For example, imagine you want to use one tool to feed many Pets, and another tool to cast many Skills. If you used each tool at a different time, you may find that (as a hypothetical example), each tool would take five minutes to run. However if you used both tools at the same time, you would find that they were both slower and each one would take ten minutes to run.

The rest of this post contains technical information about rate limiting, intended for the developers of third-party tools to help them adjust their tools to best handle rate limiting. This information has already been released to developers in the Aspiring Comrades guild.

This post was first written on 11 July 2020 and was edited after rate limiting started.

LadyAlys (talk) 23:59, July 20, 2020 (UTC)

The "Rate Limiting" section in Guidance for Comrades describes how the rate limits will be set up and gives advice about modifying tools.

The post below was made by the Staff member paglias on 2 July in the Aspiring Comrades guild. The current expected release date was changed to 20 July rather than 16 July after that post was made. A newer post with additional helpful information was on 19 July and is also copied below.

Rate Limit Announcement[]

In order to improve the stability of the server and avoid outages we've decided to implement a rate limiting mechanism for all the requests made to the public API v3 from 3rd party tools. We're currently testing the rate limits on a private server with the idea of launching it live in 2 weeks, so on the 16th of July. The limits that we're currently testing are to allow 30 requests every 60 seconds (counting from when the first request is made) for each user from 3rd party tools. The few API requests that do not require authentication will have the same limits applied to each IP address from which they're originating.
The aim of this project is to make sure that a surge in requests from a 3rd party tool won't result in degraded server performance or outages for the Habitica userbase. We've taken care to choose limits that should not impact most 3rd party tools, and which allow small bursts of requests when necessary. But if you think a 3rd party tool you're using or developing will be impacted by this change, please reach out either here or at admin@habitica.com and we'll try to figure out a solution. We're of course open to tweaking the limits if it turns out they would be too problematic, but unfortunately it won't be possible to make exceptions for individual tools or use cases.

Technical Details[]

When the rate limits go into effect on the 16th of July all responses from the API v3 will include a few extra HTTP headers:

X-RateLimit-Limit: the total amount of requests that can be made in a 60s period, will always be 30

X-RateLimit-Remaining: the number of remaining requests that can be made in the current 60s period

X-RateLimit-Reset: when the current 60s period will end

If the limit of 30 requests in a 60s period is exceeded subsequent requests made using the same USER ID for authentication or IP address will return a 429 Too Many Requests HTTP error and an extra HTTP header will be added to the response, Retry-After indicating how many seconds to wait before making a new request.

The post below was made by paglias on 19 July in the Aspiring Comrades guild.

API Update[]

In order to make life a bit easier for 3rd party tools that frequently fetch members from the API and would be impacted by the new rate limits, we're releasing a change that allows clients to fetch up to 60 members with each API call instead of 30. This can be done by using a new limit query parameter when fetching members or invites for a group or challenge. If the query parameter is not specified the old limit of 30 results per API call will be used.
More info can be found in the API docs or here. Note that the new query parameter is marked as BETA but unless an unforeseen issue comes up we don't plan to remove it.