Rate Limiting in AngelScript

Rate limiting is an important mechanism for controlling resource utilization and maintaining quality of service. AngelScript can support rate limiting with threads, queues, and timers.

#include <string>
#include <queue>
#include <chrono>
#include <thread>
#include <iostream>

void main()
{
    // First we'll look at basic rate limiting. Suppose
    // we want to limit our handling of incoming requests.
    // We'll serve these requests off a queue of the same name.
    queue<int> requests;
    for (int i = 1; i <= 5; i++)
    {
        requests.push(i);
    }

    // This limiter will receive a value every 200 milliseconds.
    // This is the regulator in our rate limiting scheme.
    auto limiter = chrono::steady_clock::now();

    // By waiting for 200 milliseconds before serving each request,
    // we limit ourselves to 1 request every 200 milliseconds.
    while (!requests.empty())
    {
        this_thread::sleep_until(limiter);
        int req = requests.front();
        requests.pop();
        cout << "request " << req << " " << chrono::system_clock::now() << endl;
        limiter += chrono::milliseconds(200);
    }

    // We may want to allow short bursts of requests in
    // our rate limiting scheme while preserving the
    // overall rate limit. We can accomplish this by
    // using a token bucket algorithm.
    const int BURST_LIMIT = 3;
    int tokens = BURST_LIMIT;
    auto burstyLimiter = chrono::steady_clock::now();

    // Every 200 milliseconds we'll try to add a new
    // token to our bucket, up to its limit of 3.
    thread tokenAdder([&]() {
        while (true)
        {
            this_thread::sleep_for(chrono::milliseconds(200));
            if (tokens < BURST_LIMIT)
            {
                tokens++;
            }
        }
    });

    // Now simulate 5 more incoming requests. The first
    // 3 of these will benefit from the burst capability
    // of our token bucket.
    queue<int> burstyRequests;
    for (int i = 1; i <= 5; i++)
    {
        burstyRequests.push(i);
    }

    while (!burstyRequests.empty())
    {
        while (tokens == 0)
        {
            this_thread::sleep_for(chrono::milliseconds(10));
        }
        tokens--;
        int req = burstyRequests.front();
        burstyRequests.pop();
        cout << "request " << req << " " << chrono::system_clock::now() << endl;
    }

    tokenAdder.detach();
}

Running our program we would see the first batch of requests handled once every ~200 milliseconds as desired.

For the second batch of requests, we would serve the first 3 immediately because of the burstable rate limiting, then serve the remaining 2 with ~200ms delays each.

This example demonstrates how to implement basic rate limiting and bursty rate limiting in AngelScript. The concepts are similar to the original, but we’ve adapted them to use AngelScript’s threading and timing facilities. We’ve used a queue instead of a channel, and a token bucket algorithm for the bursty limiter instead of a buffered channel.