Incidents and maintenance reported on status page for pxy.fi https://status.pxy.fi/ en https://status.pxy.fi/incident/1072738 Thu, 24 Sep 2026 07:48:09 -0000 https://status.pxy.fi/incident/1072738#a7990a83cb8fbc59503e5b22209980b14109c1ee88c6a8d2c388210686c240be Incident Reverse proxy health check recovered. https://status.pxy.fi/incident/1072738 Thu, 24 Sep 2026 06:44:09 -0000 https://status.pxy.fi/incident/1072738#daf0d0bea313428231b24df1ebfc517ee92682dc9bf5d1b997e9e0ddb340813f Incident Reverse proxy health check went down. https://status.pxy.fi/incident/1058374 Thu, 10 Sep 2026 14:30:09 -0000 https://status.pxy.fi/incident/1058374#b1be74fd87bba3b25c4bbf8ac36324f85c6c088218bc284eb0b22b7d4c3a37d6 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/1058374 Thu, 10 Sep 2026 13:30:13 -0000 https://status.pxy.fi/incident/1058374#97ac6615d3d2f5f6e02f0edc279f63d5dffa23a6165d805737b424270ff86916 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/1058374 Thu, 10 Sep 2026 07:30:24 -0000 https://status.pxy.fi/incident/1058374#b665f499d5a84224e3189080c4eb9878886ecc9184f22aeead7780be0a0c2966 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/1058374 Thu, 10 Sep 2026 06:00:16 -0000 https://status.pxy.fi/incident/1058374#917f34eb7a48b5e01116f021e911c264a7fc7447ba37cb6338fa361e503b418e Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/1057658 Wed, 09 Sep 2026 11:00:11 -0000 https://status.pxy.fi/incident/1057658#cd0049216edc5a5ded592b6ee4fdbc39644862be1a1efa8622046a14a66a35c4 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/1057658 Wed, 09 Sep 2026 10:00:18 -0000 https://status.pxy.fi/incident/1057658#ae8f6202848cd10f0f7a0ccacec54a76e49ebccf2de3a90ade6ffe92c74148fb Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/1048079 Thu, 03 Sep 2026 03:00:11 -0000 https://status.pxy.fi/incident/1048079#707ade74d6ee7e84582d83d19ce601afa9ef221cc33592d9e01c986c2c23c646 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/1048079 Thu, 03 Sep 2026 02:00:12 -0000 https://status.pxy.fi/incident/1048079#8cb25ea6ffebe5b55bc3fcf97bb94e0c3d973bccbb88765f112feeba8207bb32 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/1035507 Thu, 27 Aug 2026 08:00:09 -0000 https://status.pxy.fi/incident/1035507#5a2b9035d259a676c7fee0c6eca8f8754018324f00cc44082dec058ba345a502 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/1035507 Thu, 27 Aug 2026 07:00:18 -0000 https://status.pxy.fi/incident/1035507#eccdf309e47be004af2be085e8ccd2c94f87d160161cc8e2c6d0a9aff663653c Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/948192 Fri, 10 Jul 2026 05:25:00 -0000 https://status.pxy.fi/incident/948192#6b5a75c69d1dd7f9a51646320a5da7475dc471f338d059830127949560b9c9fc Incident This was resolved at: Jul 9 at 3:28pm CEST. Did not see that it had not been marked as resolved on the status page. https://status.pxy.fi/incident/948192 Thu, 09 Jul 2026 12:25:00 -0000 https://status.pxy.fi/incident/948192#d21e48a3a4265739a8f4807a0f3be274df91e3598c5f6d5aef112aac9115fd67 Incident The upstream service we proxy has some issues. - `releases.llvm.org` Will keep this open and will monitor the situation. https://status.pxy.fi/incident/948192 Thu, 09 Jul 2026 11:30:24 -0000 https://status.pxy.fi/incident/948192#88ff6afaac85380d25cfcc8acea812c353fc1ae92873f2940c37d205a21f57d9 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/895103 Wed, 13 May 2026 15:52:06 -0000 https://status.pxy.fi/incident/895103#3f33ba1e1a27e8cf245b42b5184c104152ee35b83672c879247143ed59b7c09d Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/895103 Wed, 13 May 2026 13:30:14 -0000 https://status.pxy.fi/incident/895103#eec40ecc6024fec2f82bb29def43afd21add4439e83cce87289a59a687d48dc6 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/872014 Wed, 15 Apr 2026 06:30:20 -0000 https://status.pxy.fi/incident/872014#e47c8f6eb466e588834239ce4af4a55ea22d24eb4f3c473ad0df04fa4226b9a0 Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/872014 Wed, 15 Apr 2026 05:30:19 -0000 https://status.pxy.fi/incident/872014#8ed8daccf3a7924dbe34d0afde4e53fce084f1b01ec4d0d21258828aa0e3d1e1 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/852650 Thu, 19 Mar 2026 07:30:35 -0000 https://status.pxy.fi/incident/852650#cc19cc732aefecb5cc956f4613ea2805afc3e051c7bd4070d11160ba9e740a2a Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/852650 Thu, 19 Mar 2026 06:31:01 -0000 https://status.pxy.fi/incident/852650#673b6d6deac8ade3df54ad8ed0c91e551cae2c240cb6e50494ee62705038be57 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/801472 Fri, 09 Jan 2026 18:30:22 -0000 https://status.pxy.fi/incident/801472#ddeba71af989bc09384b028712908c86a25f0373de3faa6dc0522e4c218e787f Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/801472 Fri, 09 Jan 2026 17:31:07 -0000 https://status.pxy.fi/incident/801472#88c2c9876dba4f4ba35c35bdc45ae2bd61d30e6c8bbcba958005e0511f0a0be7 Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/741242 Fri, 10 Oct 2025 14:39:51 -0000 https://status.pxy.fi/incident/741242#658ffe5ccd2177fb80c0ba1d378859246b1e81569ec902dec2a8d7f1fb9efca1 Incident Check serverless function heartbeat recovered. https://status.pxy.fi/incident/741242 Fri, 10 Oct 2025 14:33:29 -0000 https://status.pxy.fi/incident/741242#b1b2ff262267c8a24573a9e909b76cbaae1bb98a426bd46e69973e0e864e0574 Incident Check serverless function heartbeat went down. https://status.pxy.fi/incident/737636 Mon, 06 Oct 2025 15:18:36 -0000 https://status.pxy.fi/incident/737636#59ea894b00fb1557116ab48834494102d417e22298c38daa48c025e0e84c95da Incident Requesting URL able to be proxied recovered. https://status.pxy.fi/incident/737636 Mon, 06 Oct 2025 14:18:00 -0000 https://status.pxy.fi/incident/737636#73c5451c0c221a976c7283733eb87119b0e33cd950d51a80a2d2611c4b17cc6e Incident Requesting URL able to be proxied went down. https://status.pxy.fi/incident/737636 Sat, 04 Oct 2025 21:31:00 -0000 https://status.pxy.fi/incident/737636#21feb34bedf42ea67e5c7f9e63710d8a003dee9b2c558d887432009d3e5eda1b Incident Check serverless function heartbeat went down. This is paused for now https://status.pxy.fi/incident/721243 Tue, 09 Sep 2025 16:05:00 -0000 https://status.pxy.fi/incident/721243#73ebdf2c75c4ee31e6cf8a4fe40689a5d485738251ff98e67471552231f47359 Incident New release of **llvm**, `21`. Resulted in updates to: - [clang diagnostic flag matrix](https://jonasbn.github.io/til/clang/diagnostic_flags.html), the diagnostic flag for clang matrix generator - [pxy-redirect-ow-function](https://github.com/jonasbn/pxy-redirect-ow-function), the documentation proxy The following code introduced a bug: ```go if majorlevel >= "18" { ``` The Copilot explanation: > This line is a conditional statement in Go that checks whether the value of majorlevel is greater than or equal to the string "18". The comparison uses the >= operator, which, when applied to strings in Go, compares them lexicographically (dictionary order), not numerically. This means "20" >= "18" is true, but "2" >= "18" is false, because "2" comes before "18" in string order. > > Potential gotcha: If majorlevel is meant to represent a numeric value (such as a version number), comparing it as a string can lead to unexpected results. For example, "100" >= "18" is true, but "9" >= "18" is false, even though numerically 9 is less than 18. To avoid this, you should convert majorlevel to an integer before comparing: > ```go major, err := strconv.Atoi(majorlevel) if err == nil && major >= 18 { // ... } ``` > This ensures the comparison is numeric, which is usually the intended behavior for version checks or similar logic. > The code was updated and restructured to ensure proper numeric comparison for version checks and the special case handling for the newer versions. Follow up is: - Add automated tests for the proxy - Evaluate a possible random test generation strategy, so to cover more examples and potential bugs https://status.pxy.fi/incident/721245 Sat, 16 Aug 2025 16:10:00 -0000 https://status.pxy.fi/incident/721245#091f97259134853507be86f306e7c0d280485bab7096676ba09b7aa91f6e0acc Incident ## Root Cause Identified: **Connection Tracking Table Exhaustion** **Primary Issue:** Starting at approximately **19:05:58 UTC**, the system experienced a severe netfilter connection tracking table exhaustion, indicated by the kernel message: nf_conntrack: nf_conntrack: table full, dropping packet **Key Findings:** 1. **Connection Tracking Limits:** - Maximum connections allowed: **4,096** (`nf_conntrack_max`) - Current connections (now): 132 (normal level) - During the incident: Table was completely full 2. **Timeline of Events:** - **Early morning (01:03:36):** First Vector TCP connection errors appeared - **Throughout the day:** Intermittent Vector host metrics collection failures - **19:05:58:** Critical threshold reached - connection tracking table full - **19:05:58 to ~20:00:** Sustained period of packet drops and connection failures - **98 total occurrences** of "table full" messages logged 3. **Impact on Services:** - **Vector logging system:** Failed to load TCP connection info with various error patterns: - `Invalid message length: 524293` - `Invalid message length: 3012335950` - `Invalid message length: 992969696` - `Could not parse netlink response: Decode error` - **Nginx reverse proxy:** IPv6 upstream connection failures - **HTTP monitoring:** Connection closures and retry failures 4. **Attack Pattern:** - **814 SSH connection attempts** on August 16th (compared to 1,320 on August 15th) - High volume of UFW-blocked connection attempts from various IP addresses - Consistent brute-force attempts against SSH service **Trigger Analysis:** The incident was triggered by a combination of: 1. **Sustained brute-force attacks** creating many short-lived connections 2. **Vector metrics collection** attempting to read TCP connection information 3. **Normal web traffic** through the nginx reverse proxy 4. **Low connection tracking table limit** (4,096) insufficient for the traffic load **Resolution:** The incident appears to have resolved naturally as the attack volume decreased and connections timed out, freeing up space in the connection tracking table. **Recommendations:** 1. Increase `nf_conntrack_max` to handle higher connection volumes 2. Implement fail2ban or similar to automatically block brute-force attempts 3. Consider tuning connection timeout values 4. Monitor connection tracking table utilization 5. Review Vector configuration for more resilient TCP metrics collection This was a **network resource exhaustion incident** rather than a security breach, caused by the system's connection tracking table becoming overwhelmed by legitimate traffic combined with attack attempts. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:37:00 -0000 https://status.pxy.fi/incident/517170#d1da0847bef801ec88f25b9f6ab5e20cda7a6f1157c0e4b3e93a6e3e6c26f7f8 Incident Problem identified and addressed. Nginx reverse proxy was unresponsive. Analyzing and monitoring. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:01:00 -0000 https://status.pxy.fi/incident/517170#4cc29e892023e0542340d7f2dbbaa244babc659d1e733f5f032a3bfd5e2c3432 Incident The monitoring has raised an alert of unavailability. Current investigating access to: - DigitalOcean - Website - and status of domain name https://status.pxy.fi/maintenance/250742 Fri, 25 Aug 2023 19:00:00 -0000 https://status.pxy.fi/incident/250742#c8196cfca35b7e3f39055833e73c5888332864fc30b182a6bcdb136744891b72 Maintenance Updating the operating system on the reverse proxy