PII Redaction Middleware

Introduction

Note: This middleware is only available with Kipchak Enterprise.

Keeps personal data out of places it should not be: JSON responses that go to partners or front ends, and the logs that go to your log platform. Redaction is set up once in config, so no controller has to remember to do it, and a route added next year is covered on the day it ships.

• Responses: fields picked out by name or path are removed, masked (**** **** **** 4242), shortened (j***@example.com), replaced or hashed. Other responses go out byte for byte as they were.
• Logs: every record that goes through the Logger driver, from controllers, drivers and middlewares, has its message, context and extra redacted before it is written. That includes exceptions in the context.
• Detectors: emails, payment card numbers (Luhn-checked), IBANs (checksum-checked), US social security numbers, UK National Insurance numbers, international phone numbers, JWTs, bearer tokens and, optionally, IP addresses are found inside any text, with your own patterns alongside.
• Hashed pseudonyms: hash replaces a value with a keyed HMAC, the same for the same value, so support can follow one customer through the logs without seeing who they are.
• Exemptions: chosen consumers, such as an internal support console, can see responses unredacted.
• Safe defaults: passwords, secrets, tokens, API keys, card numbers and CVVs, social security and national ID numbers, passports, IBANs and dates of birth are redacted without any configuration.
• Reusable: Redaction::redact() applies the same rules to anything else, such as an audit trail or a payload sent to a third party.

Installation

To install this middleware, you need access to the Enterprise Composer repository at https://php.pkgs.1x.ax/.

composer require kipchak/middleware-redaction

Copy the sample config to config/kipchak.redaction.php and add the middleware last in middlewares/middlewares.php:

use Kipchak\Middleware\Redaction\Redaction;

// ... every other middleware first
Redaction::initialise($app);

Slim runs the last-added middleware first, so redaction then sees every response last: after error handling and after any response cache. A cached body is redacted each time it is served, and an exempt consumer's unredacted response is never cached for anyone else.

Without a config file, the defaults apply to responses and logs.

So that exceptions are redacted too, have the Error middleware log through the Logger driver (kipchak/middleware-error 1.29 or later), in config/kipchak.api.php:

'logExceptionsTo' => 'logger',

Configuration

<?php

use function Kipchak\Core\env;

return [
    'enabled' => true,

    // For the 'hash' action: at least 16 characters, kept secret. Changing it changes every pseudonym.
    'hash_key' => env('REDACTION_HASH_KEY', ''),

    // Field rules for responses and logs, added to the defaults.
    'fields' => [
        'email' => 'partial',
        'phone' => 'partial',
        'customer_ref' => 'hash',
        '**.address' => 'redact',
    ],

    // Detectors: switch one off with false, give it another action, or switch on ipv4 and ipv6.
    'detectors' => [
        'ipv4' => true,
    ],
    'custom_detectors' => [
        'employee_id' => ['pattern' => '/\bEMP-\d{6}\b/', 'action' => 'redact'],
    ],

    'responses' => [
        'enabled' => true,
        'global' => true,                    // false: attach Handlers\Redact to routes yourself
        'paths' => ['*'],
        'except_paths' => ['/health'],
        'detectors' => false,                // false, true or a list, such as ['card', 'iban']
        'fields' => [],                      // extra rules for responses only
        'unredacted_for' => ['support-console'],
        'max_body_bytes' => 10 * 1024 * 1024,
        'oversize' => 'pass',                // or 'block'
    ],

    'logs' => [
        'enabled' => true,
        'detectors' => true,
        'fields' => [],                      // extra rules for log context only
    ],
];

The config is checked when the worker starts, and a mistake names the setting.

Actions

In free text, remove and null replace the match as redact does.

Field rules

A pattern with no . matches a field of that name at any depth. Names are compared without case, spaces, _ or -, so api_key matches apiKey, API-KEY and api_key. * is a wildcard: *token matches access_token and refreshToken but not tokens_used.

A pattern with . is a path from the top of the body or log context. * is one level, list positions included, and ** is any number of levels:

'fields' => [
    'data.customer.phone' => 'partial',      // the Kipchak response envelope puts the payload under data
    'data.orders.*.card' => 'mask',
    '**.address.line1' => 'redact',
],

The first matching rule wins, and a field picked out by a rule is not looked into further. These defaults apply unless you switch them off:

Set one to false in fields to switch it off ('dob' => false), or set 'defaults' => false to start from none.

Detectors

The patterns are deliberately narrow, so order numbers, timestamps, UUIDs and local phone numbers are left alone. Detectors run in logs by default and in responses only when you switch them on, so that a customer looking at their own profile sees their own email.

Routes

To add rules for some routes only, attach the handler to them. It applies the response rules plus its own:

use Kipchak\Middleware\Redaction\Handlers\Redact;

$app->group('/v1/partners', function (RouteCollectorProxy $group) {
    // ...
})->add(new Redact($app->getContainer(), fields: ['email' => 'hash'], detectors: ['card', 'iban']));

A response redacted by a route handler is not redacted again by the global one.

Exemptions

Consumers in responses.unredacted_for get responses unredacted. Who a request is from comes from Consumer Identity, or from consumer in this config. Logs are always redacted.

Because the global handler runs before any route middleware, it can only tell who a request is from by what the request itself carries: an API key (the api_key source) or a header. To exempt consumers identified by a JWT, set responses.global to false and attach Handlers\Redact to your routes or groups after the authentication middleware, so that the token has been verified when it runs.

Using the rules elsewhere

use Kipchak\Middleware\Redaction\Redaction;

$safe = Redaction::redact($payload);                 // the log rules: field rules and detectors
$safe = Redaction::redact($payload, 'responses');    // the response rules
$text = Redaction::redactor('logs')->redactText('Call +44 20 7946 0958');

Logs are redacted by initialise(). A CLI script booted with Api::bootCli() loads no middlewares, so add the log redaction yourself:

$app = Api::bootCli();
Redaction::protectLogs($app->getContainer());

Limits

• Only application/json, text/json and +json responses are read. Other content types, and bodies sent compressed by the application, are passed through; the compressed ones with a warning in the log.
• Bodies over max_body_bytes are passed through with a warning, or refused with a 500 when oversize is 'block'.
• A redacted body is re-encoded; integers too large for PHP become strings.
• Exceptions that reach the Error middleware are written by Slim with PHP's error_log() unless kipchak.api.php has 'logExceptionsTo' => 'logger' (kipchak/middleware-error 1.29 or later), which sends them through the Logger driver and so through redaction. Set it.

Performance

Rules are compiled once per worker. A 15 KB response of 100 records takes about 0.3 ms with the default rules and 0.6 ms with every detector switched on (PHP 8.5 with OPcache and JIT). A log line with nothing to redact takes about a microsecond. Responses with nothing to redact are not re-encoded.

In testing on the Kipchak Enterprise runtime in worker mode, 6,000 concurrent requests for 100-record responses came back correctly redacted or exempt, with no personal data in any redacted response and memory steady.

Git Repository

The source code for this middleware is available on 1x.ax at https://1x.ax/mamluk/enterprise/kipchak/middlewares/redaction.