Просмотр исходного кода

docs: document server-side User-Agent blocking

Expand the web server configuration section with guidance on rejecting
crawler User-Agents at the reverse proxy or Apache layer. Document the
shipped .htaccess.disabled patterns and add a matching nginx map
example, plus notes on false positives and the official container
images.
Can Akyüz 2 недель назад
Родитель
Сommit
ab46f5d76f
1 измененных файлов с 41 добавлено и 2 удалено
  1. 41 2
      doc/Installation.md

+ 41 - 2
doc/Installation.md

@@ -141,10 +141,49 @@ you can also adjust an existing `robots.txt` file. More bot detection is
 implemented in JavaScript, but note none of these mechanisms is a 100% fail-safe
 way to prevent non-human visitors on your site.
 
+#### Server-side User-Agent blocking
+
+Client-side checks and `robots.txt` only help after a request reaches PrivateBin
+(or when a crawler chooses to honour `robots.txt`). You can additionally reject
+obvious crawler User-Agents in the web server, before PHP runs. That reduces
+noise in logs and lowers the chance that automated clients fetch paste URLs.
+
 A `.htaccess.disabled` file is provided in the root dir of PrivateBin. It blocks
 some known robots and link-scanning bots. If you use Apache, you can rename the
-file to `.htaccess` to enable this feature. If you use another webserver, you
-have to configure it manually to do the same.
+file to `.htaccess` to enable this feature. The shipped rules return HTTP 403
+when the User-Agent matches patterns such as `bot`, `spider`, `crawl`, embedded
+URLs, WhatsApp, Skype preview, or `facebookexternalhit`, while still allowing
+the Let's Encrypt validation User-Agent.
+
+If you use another web server, configure an equivalent deny list manually.
+For nginx, a `map` based on `$http_user_agent` is a common approach:
+
+```nginx
+map $http_user_agent $privatebin_bad_bot {
+    default                                                 0;
+    ~*Let's Encrypt validation server                        0;
+    ~*(?:bot|spider|crawl|https?://|WhatsApp|SkypeUriPreview|facebookexternalhit) 1;
+}
+
+server {
+    # ... existing PrivateBin server block ...
+
+    if ($privatebin_bad_bot) {
+        return 403;
+    }
+}
+```
+
+Tune the patterns for your environment. Broad matches such as `bot` can block
+legitimate tools or uptime monitors that put those substrings in their
+User-Agent. Prefer keeping health checks on simple HTTPS GETs that do not need
+to execute the client-side JavaScript bot detection.
+
+Operators of the official container images can apply the same idea in the
+image web-server config (for example nginx in
+[docker-nginx-fpm-alpine](https://github.com/PrivateBin/docker-nginx-fpm-alpine)
+or the Unit config in
+[docker-unit-alpine](https://github.com/PrivateBin/docker-unit-alpine)).
 
 ### On using Cloudflare