Incidents reported on status page for pxy.fi https://status.pxy.fi/ en https://status.pxy.fi/incident/721243 Tue, 09 Sep 2025 16:05:00 -0000 https://status.pxy.fi/incident/721243#73ebdf2c75c4ee31e6cf8a4fe40689a5d485738251ff98e67471552231f47359 New release of **llvm**, `21`. Resulted in updates to: - [clang diagnostic flag matrix](https://jonasbn.github.io/til/clang/diagnostic_flags.html), the diagnostic flag for clang matrix generator - [pxy-redirect-ow-function](https://github.com/jonasbn/pxy-redirect-ow-function), the documentation proxy The following code introduced a bug: ```go if majorlevel >= "18" { ``` The Copilot explanation: > This line is a conditional statement in Go that checks whether the value of majorlevel is greater than or equal to the string "18". The comparison uses the >= operator, which, when applied to strings in Go, compares them lexicographically (dictionary order), not numerically. This means "20" >= "18" is true, but "2" >= "18" is false, because "2" comes before "18" in string order. > > Potential gotcha: If majorlevel is meant to represent a numeric value (such as a version number), comparing it as a string can lead to unexpected results. For example, "100" >= "18" is true, but "9" >= "18" is false, even though numerically 9 is less than 18. To avoid this, you should convert majorlevel to an integer before comparing: > ```go major, err := strconv.Atoi(majorlevel) if err == nil && major >= 18 { // ... } ``` > This ensures the comparison is numeric, which is usually the intended behavior for version checks or similar logic. > The code was updated and restructured to ensure proper numeric comparison for version checks and the special case handling for the newer versions. Follow up is: - Add automated tests for the proxy - Evaluate a possible random test generation strategy, so to cover more examples and potential bugs https://status.pxy.fi/incident/721245 Sat, 16 Aug 2025 16:10:00 -0000 https://status.pxy.fi/incident/721245#091f97259134853507be86f306e7c0d280485bab7096676ba09b7aa91f6e0acc ## Root Cause Identified: **Connection Tracking Table Exhaustion** **Primary Issue:** Starting at approximately **19:05:58 UTC**, the system experienced a severe netfilter connection tracking table exhaustion, indicated by the kernel message: nf_conntrack: nf_conntrack: table full, dropping packet **Key Findings:** 1. **Connection Tracking Limits:** - Maximum connections allowed: **4,096** (`nf_conntrack_max`) - Current connections (now): 132 (normal level) - During the incident: Table was completely full 2. **Timeline of Events:** - **Early morning (01:03:36):** First Vector TCP connection errors appeared - **Throughout the day:** Intermittent Vector host metrics collection failures - **19:05:58:** Critical threshold reached - connection tracking table full - **19:05:58 to ~20:00:** Sustained period of packet drops and connection failures - **98 total occurrences** of "table full" messages logged 3. **Impact on Services:** - **Vector logging system:** Failed to load TCP connection info with various error patterns: - `Invalid message length: 524293` - `Invalid message length: 3012335950` - `Invalid message length: 992969696` - `Could not parse netlink response: Decode error` - **Nginx reverse proxy:** IPv6 upstream connection failures - **HTTP monitoring:** Connection closures and retry failures 4. **Attack Pattern:** - **814 SSH connection attempts** on August 16th (compared to 1,320 on August 15th) - High volume of UFW-blocked connection attempts from various IP addresses - Consistent brute-force attempts against SSH service **Trigger Analysis:** The incident was triggered by a combination of: 1. **Sustained brute-force attacks** creating many short-lived connections 2. **Vector metrics collection** attempting to read TCP connection information 3. **Normal web traffic** through the nginx reverse proxy 4. **Low connection tracking table limit** (4,096) insufficient for the traffic load **Resolution:** The incident appears to have resolved naturally as the attack volume decreased and connections timed out, freeing up space in the connection tracking table. **Recommendations:** 1. Increase `nf_conntrack_max` to handle higher connection volumes 2. Implement fail2ban or similar to automatically block brute-force attempts 3. Consider tuning connection timeout values 4. Monitor connection tracking table utilization 5. Review Vector configuration for more resilient TCP metrics collection This was a **network resource exhaustion incident** rather than a security breach, caused by the system's connection tracking table becoming overwhelmed by legitimate traffic combined with attack attempts. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:37:00 -0000 https://status.pxy.fi/incident/517170#d1da0847bef801ec88f25b9f6ab5e20cda7a6f1157c0e4b3e93a6e3e6c26f7f8 Problem identified and addressed. Nginx reverse proxy was unresponsive. Analyzing and monitoring. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:37:00 -0000 https://status.pxy.fi/incident/517170#d1da0847bef801ec88f25b9f6ab5e20cda7a6f1157c0e4b3e93a6e3e6c26f7f8 Problem identified and addressed. Nginx reverse proxy was unresponsive. Analyzing and monitoring. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:37:00 -0000 https://status.pxy.fi/incident/517170#d1da0847bef801ec88f25b9f6ab5e20cda7a6f1157c0e4b3e93a6e3e6c26f7f8 Problem identified and addressed. Nginx reverse proxy was unresponsive. Analyzing and monitoring. https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:01:00 -0000 https://status.pxy.fi/incident/517170#4cc29e892023e0542340d7f2dbbaa244babc659d1e733f5f032a3bfd5e2c3432 The monitoring has raised an alert of unavailability. Current investigating access to: - DigitalOcean - Website - and status of domain name https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:01:00 -0000 https://status.pxy.fi/incident/517170#4cc29e892023e0542340d7f2dbbaa244babc659d1e733f5f032a3bfd5e2c3432 The monitoring has raised an alert of unavailability. Current investigating access to: - DigitalOcean - Website - and status of domain name https://status.pxy.fi/incident/517170 Sat, 22 Feb 2025 10:01:00 -0000 https://status.pxy.fi/incident/517170#4cc29e892023e0542340d7f2dbbaa244babc659d1e733f5f032a3bfd5e2c3432 The monitoring has raised an alert of unavailability. Current investigating access to: - DigitalOcean - Website - and status of domain name https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:40:00 -0000 https://status.pxy.fi/incident/250742#ca3805a9d6d29ff7049ec1afbe7aaaef1a6b2aa131c0013e379f89c2fa687471 Maintenance completed https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:40:00 -0000 https://status.pxy.fi/incident/250742#ca3805a9d6d29ff7049ec1afbe7aaaef1a6b2aa131c0013e379f89c2fa687471 Maintenance completed https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:40:00 -0000 https://status.pxy.fi/incident/250742#ca3805a9d6d29ff7049ec1afbe7aaaef1a6b2aa131c0013e379f89c2fa687471 Maintenance completed https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:00:00 -0000 https://status.pxy.fi/incident/250742#c8196cfca35b7e3f39055833e73c5888332864fc30b182a6bcdb136744891b72 Updating the operating system on the reverse proxy https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:00:00 -0000 https://status.pxy.fi/incident/250742#c8196cfca35b7e3f39055833e73c5888332864fc30b182a6bcdb136744891b72 Updating the operating system on the reverse proxy https://status.pxy.fi/incident/250742 Fri, 25 Aug 2023 19:00:00 -0000 https://status.pxy.fi/incident/250742#c8196cfca35b7e3f39055833e73c5888332864fc30b182a6bcdb136744891b72 Updating the operating system on the reverse proxy