Guide: Centralize logs with Fluentd and Elasticsearch for PHP apps on a VPS

By Kevin Marcondes · Published on Oct 8, 2026 · Updated on Oct 16, 2026 · 9 min read

This guide shows, hands-on, how to centralize logs for PHP applications on a VPS using Fluentd and Elasticsearch. You’ll standardize JSON logs, bring up the stack with Docker Compose, configure Fluentd routing and set safe retention in Elasticsearch.

Architecture: what we'll build

The solution collects logs from PHP apps and services (Nginx/Apache), forwards them to Fluentd and indexes them in Elasticsearch. Optionally, use Kibana for queries and dashboards. Everything runs on a VPS, isolated with Docker and an internal network.

The focus is reliability and simplicity: structured JSON logs at the source, disk buffering in Fluentd, indices with automatic retention in Elasticsearch, and ports exposed only on the VPS localhost interface.

• PHP applications write logs in JSON (Monolog/PSR-3).
• Fluentd tails files and enriches metadata (host, app, env).
• Elasticsearch stores and indexes; Kibana visualizes.
• Internal Docker network for services; external ports bound to 127.0.0.1.

Prerequisites and capacity planning

You need a VPS with Docker/Compose installed, ports 9200 and 5601 reachable only locally, and enough disk for your log volume. Elasticsearch consumes RAM and I/O; tune the heap according to the machine.

Quick checklist:

- Docker and Compose installed; if needed, follow the guide How to install Docker and Docker Compose on Linux.

- Firewall configured; see the VPS Linux security checklist.

- Dedicated data directories for Elasticsearch and Fluentd.

- An index retention plan to avoid filling the disk.

Important: never expose port 9200 to the public internet without authentication and TLS. Bind to 127.0.0.1 and use an SSH tunnel or an authenticated proxy for access.

# Requisito do Elasticsearch
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elastic.conf

# Estrutura de diretórios (ajuste caminho conforme sua preferência)
mkdir -p ~/observability/elasticsearch/{data,logs}
mkdir -p ~/observability/fluentd/{state,buffer}
mkdir -p ~/observability/logs

Standardize PHP logs as JSON with context (PSR-3/Monolog)

Centralization works better with structured logs. Use Monolog with JsonFormatter and include fields such as level, message, datetime, request_id, user_id and app. This simplifies searches and correlation. See the structured JSON logs and correlation guide for modelling best practices.

Example basic configuration with Monolog for a standalone PHP project. In frameworks like Laravel, adjust config/logging.php (see docs at laravel.com).

<?php
use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Formatter\JsonFormatter;

$logDir = __DIR__ . '/logs';
if (!is_dir($logDir)) mkdir($logDir, 0775, true);

$logger = new Logger('my-php-app');
$handler = new StreamHandler($logDir . '/app.log', Logger::INFO);
$handler->setFormatter(new JsonFormatter());
$logger->pushHandler($handler);

// Contexto útil para correlação
$context = [
    'request_id' => $_SERVER['HTTP_X_REQUEST_ID'] ?? bin2hex(random_bytes(8)),
    'user_id' => $currentUserId ?? null,
    'env' => getenv('APP_ENV') ?: 'prod',
];

$logger->info('User logged in', $context);
$logger->error('Payment failed', $context + ['order_id' => 123]);

Bring up Elasticsearch and Kibana with Docker Compose

We'll start a single-node Elasticsearch and Kibana for visualization. Bind ports only on 127.0.0.1. Set the ELASTIC_PASSWORD shell variable before running. Adjust ES_JAVA_OPTS (-Xms/-Xmx) to match your RAM.

For remote access, use an SSH tunnel or an authenticated proxy. Do not expose 9200/5601 publicly.

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.4
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-changeme}
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/data:/usr/share/elasticsearch/data
      - ./elasticsearch/logs:/usr/share/elasticsearch/logs
    ports:
      - "127.0.0.1:9200:9200"
    networks:
      - observability

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.4
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD:-changeme}
    ports:
      - "127.0.0.1:5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - observability

networks:
  observability:
    name: observability

Configure Fluentd: inputs, enrichment and Elasticsearch output

Fluentd will tail the log files, enrich events with host/app/env and send them to Elasticsearch with a disk buffer. Adjust paths and tag names to your environment.

In the example we use the PHP app.log and Nginx access.log. The index prefix will be php-logs, producing daily indices.

# fluent.conf
# Input: app PHP em JSON (uma linha por evento)
<source>
  @type tail
  path /var/log/php/app.log
  pos_file /fluentd/state/app.log.pos
  tag php.app
  read_from_head true
  <parse>
    @type json
    time_key datetime
    time_type string
  </parse>
</source>

# Input: Nginx access log (texto); adapte regex conforme seu formato
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /fluentd/state/nginx.access.pos
  tag nginx.access
  read_from_head true
  <parse>
    @type regexp
    expression /^(?<remote>\S+) - (?<user>\S+) \[(?<time>[^\]]+)\] "(?<method>\S+) (?<path>\S+) (?<proto>[^"]+)" (?<status>\d{3}) (?<bytes>\d+)/
    time_key time
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</source>

# Enriquecimento: host/env/app
<filter **>
  @type record_transformer
  enable_ruby true
  <record>
    host ${Socket.gethostname}
    env ${ENV['env'] || 'prod'}
    app my-php-app
  </record>
</filter>

# Saída: Elasticsearch (índices diários com prefixo)
<match **>
  @type elasticsearch
  host elasticsearch
  port 9200
  scheme http
  user elastic
  password ${ENV['ELASTIC_PASSWORD']}
  logstash_format true
  logstash_prefix php-logs
  include_tag_key true
  reconnect_on_error true
  reload_connections false
  request_timeout 30s

  <buffer>
    @type file
    path /fluentd/buffer
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 8m
    queued_chunks_limit_size 256
    retry_forever true
    retry_max_interval 30
  </buffer>
</match>

Orchestrate Fluentd with Docker (Elasticsearch plugin)

The official Fluentd image needs the elasticsearch plugin installed. Create a small Dockerfile to install the plugin and a Compose service to run Fluentd with volumes and a shared network with Elasticsearch.

Mount the app log file into the Fluentd container as read-only. Expose the Forwarder port (24224) only if you expect logs from other hosts.

# fluentd/Dockerfile
FROM fluent/fluentd:v1.16-1
USER root
RUN gem install fluent-plugin-elasticsearch --no-document \
 && mkdir -p /fluentd/state /fluentd/buffer \
 && chown -R fluent:fluent /fluentd
COPY fluent.conf /fluentd/etc/fluent.conf
USER fluent

---
# docker-compose.fluentd.yml
version: "3.8"
services:
  fluentd:
    build: ./fluentd
    container_name: fluentd
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-changeme}
      - env=prod
    volumes:
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
      - ./logs/app.log:/var/log/php/app.log:ro
      - ./fluentd/state:/fluentd/state
      - ./fluentd/buffer:/fluentd/buffer
    depends_on:
      - elasticsearch
    networks:
      - observability
    # Descomente se for receber de outros hosts/containers via forward
    # ports:
    #   - "127.0.0.1:24224:24224"

networks:
  observability:
    external: true

Indices and retention: template + ILM in Elasticsearch

Without retention your disk will fill up. Use an ILM policy to delete old indices and an index template to apply the policy to new indices with the php-logs-* prefix. This pairs well with the daily index format Fluentd generates.

Adjust min_age to your needs and consider hot→warm phases before deletion in larger environments.

# Crie uma policy de deleção após 14 dias
curl -u elastic:${ELASTIC_PASSWORD} -X PUT \
  'http://localhost:9200/_ilm/policy/php-logs-14d' \
  -H 'Content-Type: application/json' -d '{
  "policy": {"phases": {"hot": {"actions": {}}, "delete": {"min_age": "14d", "actions": {"delete": {}}}}}
}'

# Template para índices que começam com php-logs-
curl -u elastic:${ELASTIC_PASSWORD} -X PUT \
  'http://localhost:9200/_index_template/php-logs' \
  -H 'Content-Type: application/json' -d '{
  "index_patterns": ["php-logs-*-*", "php-logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "php-logs-14d",
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings": {
      "dynamic": true,
      "properties": {
        "datetime": {"type": "date", "format": "strict_date_optional_time||epoch_millis"},
        "level": {"type": "keyword"},
        "app": {"type": "keyword"},
        "host": {"type": "keyword"},
        "message": {"type": "text"}
      }
    }
  },
  "priority": 500
}'

How to confirm it worked and next steps

Generate a test event in the app log file and check the Fluentd logs. Then search for the event in Elasticsearch. Use Kibana for dashboards and analysis. For metrics, complement with the guide to Prometheus & Grafana with Docker Compose on a VPS.

For distributed tracing in PHP, see Distributed tracing with OpenTelemetry in PHP.

# 1) Gere um log de teste (ajuste caminho)
echo '{"datetime":"2024-05-01T12:00:00Z","level":"INFO","message":"hello from php","request_id":"abc123"}' \
  >> ~/observability/logs/app.log

# 2) Veja o Fluentd entregando
docker compose -f docker-compose.fluentd.yml logs -f fluentd

# 3) Busque no Elasticsearch
curl -u elastic:${ELASTIC_PASSWORD} \
  'http://localhost:9200/php-logs-*/_search?q=message:hello&pretty'

Common troubleshooting

Some recurring issues and quick fixes to keep ingestion stable. Check permissions, formats and buffers. Tune Elasticsearch heap and Fluentd chunk_limit_size according to volume.

• Fluentd not reading file: verify path, permissions and read_from_head.
• Wrong timestamp: standardize timezone in the app and in the parser; prefer ISO 8601 with Z.
• Index mapped wrong type: fix the field in the app and recreate the index/template.
• Disk filling: reduce ILM retention, compact logs and increase rotation in the app.
• Elasticsearch slowness: tune ES_JAVA_OPTS, use shards=1 and replicas=0 for single-node.
• Nginx parser failing: adapt the regex to your real log format.

Frequently asked questions

Can I use Fluent Bit instead of Fluentd?

Yes. Fluent Bit is lighter and recommended as an agent on hosts or containers. This guide uses Fluentd for plugin flexibility and buffering. The overall architecture and Elasticsearch delivery remain the same.

How much RAM does Elasticsearch need for a single node?

It depends on volume and query complexity. For small environments, size the heap (ES_JAVA_OPTS) to a portion of available RAM and monitor. Start modest and increase as ingestion and queries grow.

How do I secure access to Elasticsearch and Kibana?

Use Elasticsearch's built-in authentication, bind to 127.0.0.1 and access via SSH tunnel or an authenticated TLS proxy. For VPS firewall and hardening, follow the VPS Linux security checklist.

Is JSON required for PHP logs?

No, but it's recommended. JSON reduces ambiguity, improves parsing and speeds up searches. If you keep free text logs, you'll need reliable parsers and accept loss of structured detail in queries.

How to handle multiple VPSs or microservices?

Run a Fluentd instance on each VPS and send to a central Elasticsearch. Use service and environment tags, and unify indices by prefix. Ensure local buffers and a retention policy to avoid data loss during network failures.

Is Kibana mandatory? Can I use another tool?

Kibana is not mandatory, but it greatly simplifies exploration and dashboards. You can query via the Elasticsearch API or integrate other compatible tools. The key is keeping indices accessible and well-mapped.

See also

• Pre-deploy checklist and quick rollback plan for PHP applications in production
• Security checklist for CI/CD pipelines with self‑hosted runners and secrets management
• Guide: Organizing PostgreSQL Migrations and Rollbacks for Small Teams
• VPS Linux security checklist before putting services into production: hardening, SSH, backups and monitoring