Nginx Source Code Notes – HTTP Core – subrequest

Subrequests are how NGINX can return the results of a request that differs from the URI that the
client sent. Depending on the configuration, they may be multiply nested and call other
subrequests. Filters can collect the responses from multiple subrequests and combine them into
one response to the client. The response is then finalized and sent to the client. Along the
way, multiple modules come into play. See http://www.aosabook.org/en/nginx.html for a detailed
explanation of NGINX internals.

A post reply from Maxim Dounin ：

Subrequests in nginx aren't really different from ordinary requests, they are handled in the
same way - with location matching and so on...

In summary, sub-requests are a crucial mechanism in Nginx’s request processing and one of the powerful features provided by Nginx. For example, through the sub-request mechanism, we can:

• Use a URL different from the original request URL to respond with the result of processing the original request;
• The system can create multiple sub-requests for the original request, or split the original request into multiple sub-requests. After the sub-requests are processed by different locations , the Postpone module merges the response results of these sub-requests and returns them to the user as the response data of the original request.
• Based on the content of the original response data, other additional data can be added to the original response. For example, the SSI (Server Include) function can be implemented using sub-requests.
• You can even treat the original response content as a URL, use subrequest to request that URL, and treat the response data as the response data of the original request.
• etc.

In Nginx, a user-triggered request is called the main request. During the processing of the main request, sub-requests can be created, and sub-requests can also create sub-subrequests. In Nginx, if request A creates sub-request B, we call A the parent request of B. In this way, these sub-requests triggered by the main request have a nested and hierarchical relationship.

Subrequests can obtain their response from any other location block. The entire process from the creation of a subrequest by the main request to Nginx starting to process it is handled entirely by Nginx through C functions, without involving network communication. For this reason, subrequests are extremely efficient.

The child’s request for “a lifetime”

Next, we’ll start with the most common use case, which is combining sub-request response data to form the final response data. We’ll analyze the code related to request creation, scheduling, and processing from a code perspective.

Create a subrequest

Nginx module code can use the function ngx_http_subrequest to create subrequests. The newly created subrequest is added to the “ready” request list of the main request and is then scheduled for processing by the function ngx_http_run_posted_request .

Creation Process

Function signature:

ngx_int_t
ngx_http_subrequest(ngx_http_request_t *r,
    ngx_str_t *uri, ngx_str_t *args, ngx_http_request_t **psr,
    ngx_http_post_subrequest_t *ps, ngx_uint_t flags);

Parameter description:

Below, we will analyze the key logic of the function ngx_http_subrequest segment by segment .

Nginx limits the number of sub-requests (including sub-sub-requests) that can be created during the processing of each main request, currently capped at 64,535. Nginx also limits the number of levels in the “sub-request relationship tree” rooted at the main request (see the next section for details), currently capped at 51 (`NGX_HTTP_MAX_SUBREQUESTS + 1`).

if (r->subrequests == 0) {
    ...
    return NGX_ERROR;
}

/*
 * 1000 is reserved for other purposes.
 */
if (r->main->count >= 65535 - 1000) {
    ...
    return NGX_ERROR;
}
...
sr->subrequests = r->subrequests - 1;

It creates a child request structure and initializes the child request based on the information carried by the parent request. For example, the child request shares the parent request’s memory pool, inherits the parent request’s request headers, and reuses the main request’s request body. However, it also has some of its own characteristics:

• Sub-requests use the HTTP GET method by default.
• Sub-requests created using the NGX_HTTP_SUBREQUEST_CLONE flag will continue the “stage processing flow” from the current “stage” of the parent request. Furthermore, the sub-request will use the same HTTP methods and location {} scope configuration as the parent request.
• Sub-requests created without using the NGX_HTTP_SUBREQUEST_CLONE flag will enter the phase processing flow from the SERVER_REWRITE phase. These sub-requests inherit the server {} scope configuration from their parent request, and during the phase processing flow, the appropriate location {} scope configuration is re-found using the URI .

sr = ngx_pcalloc(r->pool, sizeof(ngx_http_request_t));
...
c = r->connection
sr->connection = c;
...
cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module);
sr->main_conf = cscf->ctx->main_conf;
sr->srv_conf = cscf->ctx->srv_conf;
sr->loc_conf = cscf->ctx->loc_conf;

sr->pool = r->pool;
...
sr->request_body = r->request_body;
...
sr->method = NGX_HTTP_GET;
...
sr->request_line = r->request_line;
sr->uri = *uri;
...
sr->read_event_handler = ngx_http_request_empty_handler;
sr->write_event_handler = ngx_http_handler;
...
if (flags & NGX_HTTP_SUBREQUEST_CLONE) {
    sr->method = r->method;
    ...
    sr->loc_conf = r->loc_conf;
    ...
    sr->content_handler = r->content_handler;
    sr->phase_handler = r->phase_handler;
    sr->write_event_handler = ngx_http_core_run_phases;

    ngx_http_update_location_config(sr);
}

Set the sub-request attribute values ​​according to the flags parameter .

sr->subrequest_in_memory = (flags & NGX_HTTP_SUBREQUEST_IN_MEMORY) != 0;
sr->waited = (flags & NGX_HTTP_SUBREQUEST_WAITED) != 0;
sr->background = (flags & NGX_HTTP_SUBREQUEST_BACKGROUND) != 0;

Establish the relationship between subrequests and their creators, and add the subrequests to the “subrequest relationship tree.” Nginx controls the construction order of response data through the “subrequest relationship tree”: subrequest response data is sent to the requester first (before the parent request); if the parent request creates multiple subrequests, the response data is sent according to the order in which the subrequests were created. At the same time, Nginx stipulates that response data generated by “active connection requests” can be sent to the requester immediately, while response data for other requests must be sent with a delay.

• “Background subrequests” created using the NGX_HTTP_SUBREQUEST_BACKGROUND flag do not participate in the response production process, so they do not need to be added to the “subrequest relationship tree”.
• The subrequest is wrapped into a node of type ngx_http_postponed_request_t and added to the end of the “subrequest chain” ( r->postponed ).
• If the parent request is a “connection active request” and no child request has been created yet, then set the new child request to a “connection active request”.

sr->main = r->main;
sr->parent = r;
sr->post_subrequest = ps;
...
if (!sr->background) {
    if (c->data == r && r->postponed == NULL) {
        c->data = sr;
    }
    ...
    pr->request = sr;
    pr->out = NULL;
    pr->next = NULL;

    if (r->postponed) {
        for (p = r->postponed; p->next; p = p->next) { /* void */ }
        p->next = pr;

    } else {
        r->postponed = pr;
    }
}

Mark the subrequest as an internal request. Also, since the subrequest represents a new asynchronous processing branch, the reference count of the main request needs to be incremented.

sr->internal = 1;
...
sr->subrequest = r->subrequests - 1;
...
r->main->count++;

Finally, the function adds the sub-request to the main request “ready” request list, waiting to be scheduled and processed by Nginx.

return ngx_http_post_request(sr, NULL);

Sub-request relationship tree

In the previous section, we mentioned the “subrequest relationship tree” several times. In this section, we will describe how this tree-like data structure is structured and how it is used by Nginx.

To describe the sub-request relationship tree more specifically, we design a simulation scenario based on the information analyzed in the previous section, and then draw a simplified sub-request relationship diagram based on the simulation scenario to deepen our understanding.

As we discussed earlier, requests directly triggered by users are called “main requests.” Main requests can create “sub-requests,” and sub-requests can also create “sub-sub-requests.” Therefore, we will simulate the following scenario:

• The main request creates child requests “1”, “2”, and “3” in sequence. Therefore, requests “1”, “2”, and “3” are child nodes of the main request, and they are sibling nodes of each other.
• Sub-request “1” creates sub-sub-request “4”. Therefore, request “4” is a child node of request “1” and has no sibling nodes.
• Sub-request “2” creates sub-sub-requests “5” and “6” in sequence. Therefore, requests “5” and “6” are child nodes of request “2”, and they are sibling nodes of each other.
• Request “5” creates a sub-sub-request “7”. Request “7” is a child node of Request “5” and has no sibling nodes.

For the above scenario, we can simplify the ngx_http_postponed_request_t structure and the r->postponed pointer, and draw the structure diagram in Figure. a below .

If we rearrange the nodes in Figure a according to the rule of “the arrows representing parent-child relationships as the left subtree and the arrows representing sibling relationships as the right subtree” , we get a binary tree like the one in Figure b . This tree is what we call a “subtree”.

As we discussed earlier, Nginx controls the order in which response data is constructed through a “sub-request relationship tree”: sub-request response data is sent to the requester with priority (over the parent request); if the parent request creates multiple sub-requests, the response data is sent in the order in which the sub-requests were created. Therefore, we can use the inorder traversal algorithm to traverse this “sub-request relationship tree” and construct the final response data according to the traversal result.

For our hypothetical scenario, we can obtain the following sequence:

4 -> 1 -> 7 -> 5 -> 6 -> 2 -> 3 -> Main Request

It’s important to note that the sequence above does not represent the order in which Nginx processes the sub-requests. As a typical example of an event-driven concurrency model, Nginx processes all requests asynchronously, meaning the execution flow of each sub-request is unpredictable. For instance, request “3” might be processed and receive its response data much earlier than request “4”.

The sub-request relationship tree only specifies the final order in which the response results are combined. This order guarantee and combination process are handled by the ngx_http_postpone_filter_module . We will analyze this process in detail in the Sub-request Response section below .

Processing sub-requests

Once a subrequest is created, Nginx adds it to the “ready” request queue of the main request. Then, at the appropriate time, it calls the function ` ngx_http_run_posted_requests` to schedule the processing of the subrequest. This function calls the subrequest’s ` write_event_handler` to drive the subrequest processing flow.

As we know from the above, sub-requests created using the NGX_HTTP_SUBREQUEST_CLONE flag have a write_event_handler value of ngx_http_core_run_phases , and continue processing from the last phase of the parent request. Sub-requests created without this flag have a write_event_handler value of ngx_http_handler , and based on the implementation of the ngx_http_handler function , such sub-requests will begin their processing from the SERVER_REWRITE phase.

void
ngx_http_handler(ngx_http_request_t *r)
{
    ...
    if (!r->internal) {
        ...
    } else {
        cmcf = ngx_http_get_module_main_conf(r, ngx_http_core_module);
        r->phase_handler = cmcf->phase_engine.server_rewrite_index;
    }

    r->valid_location = 1;
    ...
    r->write_event_handler = ngx_http_core_run_phases;
    ngx_http_core_run_phases(r);
}

Next, Nginx begins processing the sub-requests just like it would the main request. This part of the process will not be detailed here.

In our analysis of the function ngx_http_finalize_request, we explained how the parent request is woken up after the child request has finished processing. For a detailed analysis, please refer to…here Found it. However, there is a key point that is helpful for understanding the following content, which we will reiterate here: After Nginx completes the entire process of a sub-request, it will set the parent request of the sub-request as the “active connection request” for all its connections.

Sub-request response

The sub-request response data combination is handled by the ngx_http_postpone_filter_module .

As mentioned earlier, due to Nginx’s asynchronous mechanism for handling sub-requests, the order in which sub-requests in the “sub-request relationship tree” receive their responses is uncertain. If a sub-request that has already been processed is not a “connection-active request,” then this module is responsible for caching the response data and sending it out at an appropriate time. Simultaneously, this module must also select the next request as the “connection-active request” after the response data for the “connection-active request” has been sent.

Next, we will analyze the main processing function of this module , ngx_http_postpone_filter , in detail :

If the current request is not an “active connection request”, we need to temporarily cache its response data. The function ngx_http_postpone_filter_add appends the response data to the tail of the current request’s r->postponed singly linked list. It can be seen that r->postponed not only stores the child requests that have not been processed, but also the response data that the request itself has not yet sent.

if (r != c->data) {

    if (in) {
        ngx_http_postpone_filter_add(r, in);
        return NGX_OK;
    }

    return NGX_OK;
}

If the current request is an “active connection request” and it has no subrequests or has not sent response data, then the response data for this call can be sent directly (to the next body filter).

if (r->postponed == NULL) {

    if (in || c->buffered) {
        return ngx_http_next_body_filter(r->main, in);
    }

    return NGX_OK;
}

If the current request is an “active connection request”, but it still has unprocessed sub-requests or cached response data from the previous call, then we will append the response data of this call to the tail of the r->postponed singly linked list.

if (in) {
    ngx_http_postpone_filter_add(r, in);
}

Next, select one of the sub-requests of the current request as the “connection active request”, or pass the temporary response data to the next filter module.

• If a subrequest selected as a “connection-active request” has sub-subrequests, then when the `ngx_http_postpost_filter` function processes that subrequest, it will also set that sub-subrequest as a “connection-active request,” and so on. This ensures the correct order in which subrequest responses are sent.
• If the current request has response data, it must be sent after its sub-requests have been processed.

do {
    pr = r->postponed;

    if (pr->request) {
        ...
        r->postponed = pr->next;
        c->data = pr->request;

        return ngx_http_post_request(pr->request, NULL);
    }

    if (pr->out == NULL) {
        ...
    } else {
        ...
        if (ngx_http_next_body_filter(r->main, pr->out) == NGX_ERROR) {
            return NGX_ERROR;
        }
    }

    r->postponed = pr->next;

} while (r->postponed);

In this way, Nginx, under the asynchronous model, coordinates the work of the functions ngx_http_finalize_request and ngx_http_postpone_filter to achieve the sequential sending of requests and responses in the “sub-request relationship tree”.

Another noteworthy point is that the subrequest response also consists of a response header and a response body (after all, Nginx treats subrequests as normal requests). The response header is generally not sent to the user (the main request response header is sent to the user), so most header filter modules simply discard the subrequest response header. For example, the ` ngx_http_header_filter` function in the ` ngx_http_header_filter_module` handles it as follows:

void
ngx_http_header_filter(ngx_http_request_t *r)
{
    ...
    if (r != r->main) {
        return NGX_OK;
    }
    ...
}

Other application scenarios

In addition to using sub-request response data to combine the final response ( ngx_http_addition_filter_module ), there are many other application scenarios in various Nginx modules .

For example, the module ngx_http_auth_request_module uses sub-requests to verify user permissions to determine whether to continue processing the main request; the module ngx_http_ssi_filter_module creates sub-requests based on the Server Include directive in the main request response content and concatenates its response with the original response; the module ngx_http_mirror_module uses the creation of “background sub-requests” to implement traffic replication, and so on.

The `ngx_http_mirror_module` is a new module introduced in Nginx 1.13.4. It allows for easy replication of real online requests to perform stress tests on modules not yet deployed. This eliminates the need for more complex tools like `tcpcopy` and `goreplay` in systems using the HTTP protocol to achieve similar functionality. We will analyze the code for this module in the next article.

Don’t leave me so easily, please leave something behind…