diff --git a/.gitignore b/.gitignore index f6ad211..2c695bd 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ vendor/ composer.lock +.phpunit.result.cache diff --git a/README.md b/README.md index 05017eb..09d149b 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,42 @@ $headers = [ $response = $http->head('http://example.com/', $headers); ``` +### Safe mode + +When the URLs you fetch come from other people (a user's website, a +discovered endpoint, a link in a post), turn on safe mode so they cannot +point your server at itself or at your private network: + +```php +$http = new p3k\HTTP(); +$http->set_safe_mode(true); +``` + +In safe mode: + +* only `http` and `https` URLs are fetched (no `file`, `gopher`, `dict`, …) +* the host must resolve only to public addresses; loopback, private, link-local + and other reserved ranges are refused, as are hosts written as unusual IP + forms like `2130706433` or `0x7f.1` +* curl connects to exactly the addresses that were checked, so a DNS answer + that changes between the check and the request makes no difference +* redirects are followed one at a time and every hop is checked the same way; + `Authorization`, `Proxy-Authorization` and `Cookie` headers are dropped when a + redirect goes to a different origin + +A refused request makes no connection and returns `code` 0 with `error` set to +`blocked_url` (or `dns_error` when the host does not resolve). + +To reach a private server on purpose, such as a development site on your LAN, +allow it by hostname, address or CIDR range: + +```php +$http->set_safe_mode(true, ['dev.example.com', '10.11.11.0/24']); +``` + +Custom transports get the same checks. A transport that implements +`p3k\HTTP\Pinnable` is also held to the checked addresses, as the curl transport is. + ### Response The get/post/head functions will return an array with the following properties: diff --git a/composer.json b/composer.json index 48f421c..b5550de 100644 --- a/composer.json +++ b/composer.json @@ -20,5 +20,14 @@ } }, "autoload-dev": { + "psr-4": { + "p3k\\HTTP\\Tests\\": "tests/" + } + }, + "require-dev": { + "phpunit/phpunit": "^10 || ^11" + }, + "scripts": { + "test": "phpunit" } } diff --git a/phpunit.xml b/phpunit.xml new file mode 100644 index 0000000..495915a --- /dev/null +++ b/phpunit.xml @@ -0,0 +1,8 @@ + + + + + tests + + + diff --git a/src/p3k/HTTP/Curl.php b/src/p3k/HTTP/Curl.php index 8b25774..75cfbf2 100644 --- a/src/p3k/HTTP/Curl.php +++ b/src/p3k/HTTP/Curl.php @@ -1,10 +1,11 @@ _timeout = $timeout; } + public function pin_addresses($resolve) { + $this->_pinned = $resolve; + } + public function get($url, $headers=[]) { $ch = curl_init($url); $this->_set_curlopts($ch, $url); @@ -103,6 +108,17 @@ class Curl implements Transport { curl_setopt($ch, CURLOPT_TIMEOUT_MS, round($this->_timeout * 1000)); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 2000); curl_setopt($ch, CURLOPT_HTTP_VERSION, $this->_http_version()); + if($this->_pinned !== null) { + if(defined('CURLOPT_PROTOCOLS_STR')) { + curl_setopt($ch, CURLOPT_PROTOCOLS_STR, 'http,https'); + curl_setopt($ch, CURLOPT_REDIR_PROTOCOLS_STR, 'http,https'); + } else { + curl_setopt($ch, CURLOPT_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS); + curl_setopt($ch, CURLOPT_REDIR_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS); + } + curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); + curl_setopt($ch, CURLOPT_RESOLVE, $this->_pinned); + } } private function _http_version() { diff --git a/src/p3k/HTTP/Guard.php b/src/p3k/HTTP/Guard.php new file mode 100644 index 0000000..cb7e05c --- /dev/null +++ b/src/p3k/HTTP/Guard.php @@ -0,0 +1,144 @@ +_allow_cidrs[] = strpos($entry, '/') !== false ? $entry : trim($entry, '[]') . (strpos($entry, ':') !== false ? '/128' : '/32'); + else + $this->_allow_hosts[] = rtrim($entry, '.'); + } + $this->_resolver = $resolver ?: [self::class, 'resolve']; + } + + /** + * @return array Either ['host' => ..., 'port' => ..., 'addresses' => [...]] + * or ['error' => 'blocked_url'|'dns_error', 'error_description' => ...] + */ + public function check($url) { + $parts = parse_url($url); + if($parts === false || !isset($parts['scheme']) || !isset($parts['host']) || $parts['host'] === '') + return self::_error('blocked_url', 'The URL is not a valid absolute URL'); + + $scheme = strtolower($parts['scheme']); + if($scheme !== 'http' && $scheme !== 'https') + return self::_error('blocked_url', 'Only http and https URLs can be fetched'); + + $host = strtolower($parts['host']); + $port = isset($parts['port']) ? (int)$parts['port'] : ($scheme === 'https' ? 443 : 80); + + if($host[0] === '[') { + $literal = substr($host, 1, -1); + if(!filter_var($literal, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6)) + return self::_error('blocked_url', 'The URL has an invalid IPv6 address'); + $addresses = [$literal]; + } elseif(filter_var($host, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4)) { + $addresses = [$host]; + } elseif(preg_match(self::NUMERIC_HOST, $host)) { + return self::_error('blocked_url', 'The URL\'s host is an IP address in an unusual form'); + } else { + $addresses = call_user_func($this->_resolver, rtrim($host, '.')); + if(!$addresses) + return self::_error('dns_error', 'Could not resolve ' . $host); + } + + $hostAllowed = in_array(rtrim($host, '.'), $this->_allow_hosts, true); + foreach($addresses as $address) { + if(!$hostAllowed && !$this->_address_allowed($address)) + return self::_error('blocked_url', $host . ' resolves to ' . $address . ', which is not a public address'); + } + + return ['host' => $host, 'port' => $port, 'addresses' => array_values($addresses)]; + } + + /** Every IPv4 and IPv6 address the system resolver knows for a host. */ + public static function resolve($host) { + $addresses = gethostbynamel($host) ?: []; + $records = @dns_get_record($host, DNS_AAAA); + foreach($records ?: [] as $record) { + if(isset($record['ipv6'])) + $addresses[] = $record['ipv6']; + } + return array_values(array_unique($addresses)); + } + + /** Whether an address is on the public internet, looking inside IPv6 forms that carry an IPv4 address. */ + public static function is_public($address) { + if(!filter_var($address, FILTER_VALIDATE_IP, FILTER_FLAG_GLOBAL_RANGE)) + return false; + + $embedded = self::_embedded_ipv4($address); + if($embedded !== null) + return self::is_public($embedded); + + return true; + } + + public static function in_cidr($address, $cidr) { + list($subnet, $bits) = array_pad(explode('/', $cidr, 2), 2, null); + $a = @inet_pton($address); + $s = @inet_pton($subnet); + if($a === false || $s === false || strlen($a) !== strlen($s)) + return false; + $bits = $bits === null ? strlen($a) * 8 : (int)$bits; + $bytes = intdiv($bits, 8); + if(substr($a, 0, $bytes) !== substr($s, 0, $bytes)) + return false; + $rest = $bits % 8; + if($rest === 0) + return true; + $mask = chr((0xff << (8 - $rest)) & 0xff); + return ($a[$bytes] & $mask) === ($s[$bytes] & $mask); + } + + private function _address_allowed($address) { + foreach($this->_allow_cidrs as $cidr) { + if(self::in_cidr($address, $cidr)) + return true; + } + return self::is_public($address); + } + + // NAT64 (64:ff9b::/96, 64:ff9b:1::/48) and 6to4 (2002::/16) addresses + // reach an IPv4 address, which must be public too. + private static function _embedded_ipv4($address) { + $bin = @inet_pton($address); + if($bin === false || strlen($bin) !== 16) + return null; + if(self::in_cidr($address, '64:ff9b::/96') || self::in_cidr($address, '64:ff9b:1::/48')) + return inet_ntop(substr($bin, 12, 4)); + if(self::in_cidr($address, '2002::/16')) + return inet_ntop(substr($bin, 2, 4)); + return null; + } + + private static function _error($code, $description) { + return ['error' => $code, 'error_description' => $description]; + } +} diff --git a/src/p3k/HTTP/Pinnable.php b/src/p3k/HTTP/Pinnable.php new file mode 100644 index 0000000..3c5b423 --- /dev/null +++ b/src/p3k/HTTP/Pinnable.php @@ -0,0 +1,17 @@ + ['file', '/dev/null', 'w'], 2 => ['file', '/dev/null', 'w']], + $pipes + ); + for($i = 0; $i < 50; $i++) { + $socket = @fsockopen('127.0.0.1', self::$port); + if($socket) { fclose($socket); return; } + usleep(100000); + } + self::fail('The test server did not start'); + } + + public static function tearDownAfterClass(): void { + proc_terminate(self::$server); + } + + private function http() { + $http = new HTTP('test'); + // "pinned.example" exists only in this resolver; curl can reach it only + // through the pinned address. + $http->set_safe_mode(true, ['127.0.0.1'], function($host) { return $host === 'pinned.example' ? ['127.0.0.1'] : []; }); + return $http; + } + + public function testConnectsToThePinnedAddress() { + $response = $this->http()->get('http://pinned.example:' . self::$port . '/'); + $this->assertSame(200, $response['code']); + $this->assertSame('host=pinned.example:' . self::$port, $response['body']); + } + + public function testRedirectsAreCheckedHopByHop() { + $port = self::$port; + $response = $this->http()->get("http://pinned.example:$port/?to=" . rawurlencode("http://127.0.0.1:$port/")); + $this->assertSame(200, $response['code']); + $this->assertSame("http://127.0.0.1:$port/", $response['url']); + + $response = $this->http()->get("http://pinned.example:$port/?to=" . rawurlencode("http://127.0.0.2:$port/")); + $this->assertSame('blocked_url', $response['error']); + } + + public function testPinnedCurlRefusesOtherProtocols() { + $curl = new Curl(); + $curl->pin_addresses([]); + $response = $curl->get('dict://127.0.0.1:' . self::$port . '/info'); + $this->assertSame(0, $response['code']); + $this->assertNotSame('', $response['error_description']); + } +} diff --git a/tests/GuardTest.php b/tests/GuardTest.php new file mode 100644 index 0000000..c953cd7 --- /dev/null +++ b/tests/GuardTest.php @@ -0,0 +1,91 @@ +assertTrue(Guard::is_public($address)); + } + + #[\PHPUnit\Framework\Attributes\DataProvider('nonPublicAddresses')] + public function testNonPublicAddresses($address) { + $this->assertFalse(Guard::is_public($address)); + } + + public static function blockedUrls() { + return [ + ['gopher://127.0.0.1:6379/_SET%20x%201'], ['dict://127.0.0.1:6379/info'], ['file:///etc/passwd'], + ['ftp://example.com/'], ['ldap://example.com/'], ['javascript:alert(1)'], ['/relative'], ['https://'], + ['http://127.0.0.1/'], ['http://localhost.example/'], ['http://[::1]/'], ['http://[::ffff:127.0.0.1]/'], + ['http://2130706433/'], ['http://0x7f000001/'], ['http://0177.0.0.1/'], ['http://127.1/'], ['http://0/'], + ['http://169.254.169.254/latest/meta-data/'], ['http://mixed.example/'], + ]; + } + + #[\PHPUnit\Framework\Attributes\DataProvider('blockedUrls')] + public function testBlockedUrls($url) { + $result = self::guard(['localhost.example' => ['127.0.0.1'], 'mixed.example' => ['93.184.216.34', '10.0.0.1']])->check($url); + $this->assertSame('blocked_url', $result['error'] ?? null, $url); + } + + public function testPublicHostIsAllowedWithItsAddresses() { + $result = self::guard(['example.com' => ['93.184.216.34', '2606:2800:220:1::']])->check('https://Example.com/path'); + $this->assertSame(['host' => 'example.com', 'port' => 443, 'addresses' => ['93.184.216.34', '2606:2800:220:1::']], $result); + } + + public function testExplicitPort() { + $result = self::guard(['example.com' => ['93.184.216.34']])->check('http://example.com:8080/'); + $this->assertSame(8080, $result['port']); + } + + public function testUnresolvableHost() { + $this->assertSame('dns_error', self::guard()->check('https://nowhere.example/')['error']); + } + + public function testAllowedHost() { + $guard = self::guard(['dev.example' => ['10.11.11.80']], ['dev.example']); + $this->assertSame(['10.11.11.80'], $guard->check('https://dev.example/')['addresses']); + $this->assertArrayHasKey('error', $guard->check('https://other.example/')); + } + + public function testAllowedRange() { + $guard = self::guard(['dev.example' => ['10.11.11.80'], 'db.example' => ['10.11.12.5']], ['10.11.11.0/24']); + $this->assertArrayNotHasKey('error', $guard->check('https://dev.example/')); + $this->assertArrayHasKey('error', $guard->check('https://db.example/')); + } + + public function testAllowedSingleAddress() { + $guard = self::guard([], ['127.0.0.1', '::1']); + $this->assertArrayNotHasKey('error', $guard->check('http://127.0.0.1:8000/')); + $this->assertArrayNotHasKey('error', $guard->check('http://[::1]:8000/')); + $this->assertArrayHasKey('error', $guard->check('http://127.0.0.2/')); + } + + public function testInCidr() { + $this->assertTrue(Guard::in_cidr('10.11.11.80', '10.11.11.0/24')); + $this->assertFalse(Guard::in_cidr('10.11.12.80', '10.11.11.0/24')); + $this->assertTrue(Guard::in_cidr('10.11.11.80', '10.8.0.0/13')); + $this->assertTrue(Guard::in_cidr('fd00::1', 'fc00::/7')); + $this->assertFalse(Guard::in_cidr('10.0.0.1', 'fc00::/7')); + } +} diff --git a/tests/RecordingTransport.php b/tests/RecordingTransport.php new file mode 100644 index 0000000..1de2ea4 --- /dev/null +++ b/tests/RecordingTransport.php @@ -0,0 +1,43 @@ + [code, headers string, body] */ + public function __construct(array $responses) { + $this->responses = $responses; + } + + public function pin_addresses($resolve) { $this->pinned = $resolve; } + public function set_timeout($timeout) {} + public function set_max_redirects($max) { $this->max_redirects = $max; } + + public function get($url, $headers=[]) { return $this->answer('GET', $url, false, $headers); } + public function post($url, $body, $headers=[]) { return $this->answer('POST', $url, $body, $headers); } + public function put($url, $body, $headers=[]) { return $this->answer('PUT', $url, $body, $headers); } + public function head($url, $headers=[]) { return $this->answer('HEAD', $url, false, $headers); } + + private function answer($method, $url, $body, $headers) { + $this->requests[] = ['method' => $method, 'url' => $url, 'body' => $body, 'headers' => $headers, 'pinned' => $this->pinned]; + list($code, $header, $responseBody) = $this->responses[$url] ?? [404, '', 'not found']; + return [ + 'code' => $code, + 'header' => "HTTP/1.1 $code X\r\n" . $header, + 'body' => $responseBody, + 'error' => '', + 'error_description' => '', + 'url' => $url, + 'debug' => '', + ]; + } +} diff --git a/tests/SafeModeTest.php b/tests/SafeModeTest.php new file mode 100644 index 0000000..22dec96 --- /dev/null +++ b/tests/SafeModeTest.php @@ -0,0 +1,99 @@ + ['93.184.216.34'], + 'b.example' => ['93.184.216.35', '2606:2800:220:1::'], + 'internal.example' => ['10.0.0.5'], + ]; + + private function http(array $responses, &$transport) { + $transport = new RecordingTransport($responses); + $http = new HTTP('test', $transport); + $http->set_safe_mode(true, [], function($host) { return self::DNS[$host] ?? []; }); + return $http; + } + + public function testBlockedUrlMakesNoRequest() { + $http = $this->http([], $transport); + foreach(['gopher://127.0.0.1:6379/_FLUSHALL', 'file:///etc/passwd', 'http://127.0.0.1/', 'https://internal.example/'] as $url) { + $response = $http->post($url, 'x'); + $this->assertSame('blocked_url', $response['error'], $url); + $this->assertSame(0, $response['code']); + } + $this->assertSame([], $transport->requests); + } + + public function testRequestIsPinnedToCheckedAddresses() { + $http = $this->http(['https://b.example/' => [200, '', 'ok']], $transport); + $response = $http->get('https://b.example/'); + $this->assertSame(200, $response['code']); + $this->assertSame(['b.example:443:93.184.216.35', 'b.example:443:[2606:2800:220:1::]'], $transport->requests[0]['pinned']); + $this->assertSame(0, $transport->max_redirects); + } + + public function testRedirectToPrivateAddressIsRefused() { + $http = $this->http([ + 'https://a.example/' => [302, "Location: http://169.254.169.254/latest/meta-data/\r\n", ''], + ], $transport); + $response = $http->get('https://a.example/'); + $this->assertSame('blocked_url', $response['error']); + $this->assertCount(1, $transport->requests); + } + + public function testRedirectToOtherSchemeIsRefused() { + $http = $this->http([ + 'https://a.example/token' => [307, "Location: gopher://a.example:6379/_x\r\n", ''], + ], $transport); + $this->assertSame('blocked_url', $http->post('https://a.example/token', 'code=1')['error']); + $this->assertCount(1, $transport->requests); + } + + public function testRedirectsAreFollowedKeepingMethodAndBody() { + $http = $this->http([ + 'https://a.example/one' => [301, "Location: /two\r\n", ''], + 'https://a.example/two' => [200, "Content-Type: text/plain\r\n", 'done'], + ], $transport); + $response = $http->post('https://a.example/one', 'body', ['Authorization: Bearer secret']); + $this->assertSame(200, $response['code']); + $this->assertSame('done', $response['body']); + $this->assertSame('https://a.example/two', $response['url']); + $this->assertSame('POST', $transport->requests[1]['method']); + $this->assertSame('body', $transport->requests[1]['body']); + $this->assertContains('Authorization: Bearer secret', $transport->requests[1]['headers']); + } + + public function testCredentialsAreDroppedWhenARedirectLeavesTheOrigin() { + $http = $this->http([ + 'https://a.example/' => [302, "Location: https://b.example/\r\n", ''], + 'https://b.example/' => [200, '', 'ok'], + ], $transport); + $http->get('https://a.example/', ['Authorization: Bearer secret', 'Cookie: a=b', 'Accept: text/html']); + $this->assertSame(['Accept: text/html', 'User-Agent: test'], $transport->requests[1]['headers']); + } + + public function testTooManyRedirects() { + $http = $this->http([ + 'https://a.example/loop' => [302, "Location: /loop\r\n", ''], + ], $transport); + $http->set_max_redirects(3); + $response = $http->get('https://a.example/loop'); + $this->assertSame('too_many_redirects', $response['error']); + $this->assertCount(4, $transport->requests); + } + + public function testOffByDefault() { + $transport = new RecordingTransport([]); + $http = new HTTP('test', $transport); + $this->assertFalse($http->safe_mode()); + $http->get('http://127.0.0.1/'); + $this->assertCount(1, $transport->requests); + $this->assertNull($transport->requests[0]['pinned']); + $this->assertSame(8, $transport->max_redirects); + } +} diff --git a/tests/server/index.php b/tests/server/index.php new file mode 100644 index 0000000..f8477ec --- /dev/null +++ b/tests/server/index.php @@ -0,0 +1,7 @@ +