Apache RewriteRule discards SetInputFilter DEFLATE config directive

Question

I have the following (simplified) folder/file structure:

/.htaccess
/test.php
/api/web/index.php

And the following directive in apache config:

<IfModule mod_deflate.c>
    <IfModule mod_filter.c>
        SetInputFilter DEFLATE
   </IfModule>
</IfModule>

I am sending a POST request with a gzipped body with the appropiated headers:

POST /test.php HTTP/1.1
Host: 192.168.1.248
Authorization: Bearer ed717c077e4bf81201196011adb457731b24e19d
Content-Type: application/json
Content-Encoding: gzip

And I have the following config for the .htaccess file:

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^api/(.*) api/web/index.php/$1 [NC,L]

The issue is, if I post to /test.php, everything works as expected, the body is deflated and I can access decompressed contents just right.

However if I post to something that gets redirected (/api/ or /api/v1/project) the index.php script does not get the body decompressed.

I think it must be related to the RewriteRule directive ignoring the SetInputFilter directive, but, how can I avoid this situation?

I tried to add the SetInputFilter directive directly in the .htaccess without solving the issue (may be it was not in the right place?).

Do you know how can I solve this?

Answer 1

Indeed, there is a problem. The first thing I did to investigate deeper was to log traces of concerned modules (rewrite, filter and deflate).

mod_rewrite logs were OK, nothing suspicious there. To make sure everything was really OK, I looked at the very last version of its source code. And again, nothing suspicious regarding encoding/decoding (nor http request/response headers, more generally).

So I started thinking the problem may come from either filter or deflate module, even if it may also come from somewhere else. To confirm/infirm what I thought, I looked at those modules logs. Quickly, I was able to see a difference between the two test cases: with or without mod_rewrite involved.

mod_rewrite not involved

mod_deflate.c(1421): [client 127.0.0.1:53000] AH01393: Zlib: Inflated 35 to 41 : URL /test.php
mod_filter.c(188): [client 127.0.0.1:53000] Content-Type condition for 'deflate' matched

I took this one as a reference to compare the next case below

mod_rewrite involved

mod_filter.c(188): [client 127.0.0.1:53002] Content-Type condition for 'deflate' matched

Interesting. Actually, it looked like mod_deflate was the problem. I suspected its action was after the right moment. That's the reason why you don't see it in action, here, in this case.

Solution

So far, so good. So... what ? Well, a quick search on the known bugs list of Apache, with keywords mod_deflate too late, gave me by chance what I was searching for. This ticket called mod_deflate adjusts the headers "too late", states the following:

When mod_deflate is used to inflate, it must adjust the request headers (e.g. it needs to remove the "Content-Length" header and adjust the "Content-Encoding" header).

Currently mod_deflate adjusts the headers when the request body is read. But this is too late. For example, if a content generator module needs to look at the request headers before reading the request body, the content generator module "sees" the old (unmodified) headers.

mod_deflate should adjust the headers in an early stage, for example in a fixup hook (ap_hook_fixups).

Eureka ! This is exactly the problem we're facing. Now, good news is there is a patch to overcome this issue. Bad news: it is not yet reviewed/accepted/merged in available versions.

You have the choice:

1. Apply this patch and recompile your server. It should work because the all thing makes sense. But, be careful... this may introduce other bugs/holes (that's sometimes the case, even when reviewed/accepted)
2. Wait for it to be included in available versions (maybe a long time, considering the ticket date). By then, use your custom deflate with php.

Update

Just tried to apply the patch and recompile mod_deflate. Looks like it's on the right track: it eats Content-Encoding header. Anyway, Content-Length is still there. Result: no yet decompressed. So, there is still something to do and adapt, but the problem is definitely in that area.

Update 2 (working)

I managed to make it work, finally. Here is the patch I applied to Apache (httpd version 2.4.34):

diff --git a/modules/filters/mod_deflate.c b/modules/filters/mod_deflate.c
index 1428460..cc8c0cb 100644
--- a/modules/filters/mod_deflate.c
+++ b/modules/filters/mod_deflate.c
@@ -1099,10 +1099,10 @@ static apr_status_t deflate_in_filter(ap_filter_t *f,

         if (!ctx) {
             /* only work on main request/no subrequests */
-            if (!ap_is_initial_req(r)) {
+            /*if (!ap_is_initial_req(r)) {
                 ap_remove_input_filter(f);
                 return ap_get_brigade(f->next, bb, mode, block, readbytes);
-            }
+            }*/

             /* We can't operate on Content-Ranges */
             if (apr_table_get(r->headers_in, "Content-Range") != NULL) {

Actually, I made mod_deflate handle sub-requests too. I'm not sure it's not gonna break some other modules, but it works for your use case (it's more a proof of concept). Anyway, I proposed my patch on the ticket mentioned above. Here is a screenshot of the result: