tag:status.patchman.co,2005:/historyPatchman Status - Incident History2024-03-19T09:48:33+01:00Patchmantag:status.patchman.co,2005:Incident/202225572024-03-12T15:47:42+01:002024-03-12T15:47:42+01:00Intermittent HTTP 500 errors<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>15:47</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>14:14</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:23</var> CET</small><br><strong>Update</strong> - We are now seeing that the issue also affected the Customer API with HTTP 400 errors. If you have such errors in your logs, they will be related to this problem, but should now no longer appear.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:10</var> CET</small><br><strong>Identified</strong> - We have implemented a temporary fix while we develop a permanent one. The issue should no longer show, although we do not consider the matter resolved for now.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:00</var> CET</small><br><strong>Investigating</strong> - We are seeing intermittent HTTP 500 errors specifically for logged in user sessions on the webinterface of the Patchman Portal. We are investigating the issue.</p>tag:status.patchman.co,2005:Incident/197921192024-01-23T09:30:07+01:002024-01-23T09:30:07+01:00Customer API authentication errors<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>09:30</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>17:46</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>11:05</var> CET</small><br><strong>Update</strong> - Authentication problems for existing keys have been resolved, but the problem remains for now on newly generated API credentials.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:42</var> CET</small><br><strong>Identified</strong> - We have identified a bug in API authentication on the Customer API that we previously thought was related to the large outage of last week, but has now been determined to be a separate issue. We are developing a fix for this problem and will have this rolled out later today.</p>tag:status.patchman.co,2005:Incident/197519432024-01-22T10:38:53+01:002024-01-22T10:38:53+01:00Sustained HTTP 500 errors on Portal<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:38</var> CET</small><br><strong>Resolved</strong> - We have seen no further regressions. While of course we continue to monitor diligently, we will close this incident for now. We will be working on a post-mortem for this incident soon.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>14:15</var> CET</small><br><strong>Monitoring</strong> - At this time we are seeing that all errors have been resolved. We will continue to monitor the situation for several days. If you do encounter anything you are unsure about, please don't hesitate to reach out to support@patchman.co.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>09:20</var> CET</small><br><strong>Update</strong> - There are still certain errors related to API authentication and key management. We are aware of the cause and are working on its resolution.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>09:13</var> CET</small><br><strong>Update</strong> - All systems are operational again, although the Data Processing Infrastructure is still processing a significant backlog. It may take some time for the data to be up-to-date again in the Portal.<br /><br />We are continuing work on the backend to run additional checks, verifications and add further redundancies.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>09:07</var> CET</small><br><strong>Update</strong> - The restoration has been completed, and we are bringing our services back up again.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>19:49</var> CET</small><br><strong>Update</strong> - We are currently restoring a backup from cold storage, which we expect to take several hours to complete. We will update this post once there is progress to report on.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>18:43</var> CET</small><br><strong>Update</strong> - Our restoration process has hit a problem, and we have to re-evaluate the next steps. There are still fallback plans, but we have to validate the steps we have before we can definitively decide on which to use.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:58</var> CET</small><br><strong>Update</strong> - We are seeing that another one of our redundancy steps is taking longer than expected. While we understand that it is frustrating to not have reliable estimations for the system coming back online, we are doing everything we can to get this done as soon as humanly possible, and are at this point simply bottlenecked by systems having to process the large amounts of data that we hold in this affected database.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:25</var> CET</small><br><strong>Update</strong> - The recovery process is proceeding steadily, but we have hit a delay because one step took longer than expected. Our predicted online time is now 18:15 CET (UTC+1).</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:08</var> CET</small><br><strong>Update</strong> - A viable path to restoration has been identified, and we are currently going through the steps of making this happen. We have various redundant steps and failsafes in this procedure that give us the best chance possible, but all these additional steps mean that the process will take some time to complete. At this point, we expect the remaining downtime to last until roughly 17:30 CET (UTC+1).</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>15:04</var> CET</small><br><strong>Identified</strong> - We have concluded that unfortunately part of our main database system for the Patchman Portal has been corrupted. We are currently trying to determine the safest course of action for remediation between cluster recovery and backup restoration. Our goal is of course to minimize data loss, but we want to make sure we do not use a solution that does not fully remediate the problem of corruption we are facing.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>14:14</var> CET</small><br><strong>Update</strong> - Our determination so far is a consistency problem in our database backend. We are trying to determine the exact problem so that we can apply the correct resolution. For now, to protect your data in our system, we will unfortunately need to keep the Portal down.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>13:54</var> CET</small><br><strong>Investigating</strong> - We are currently investigating an issue that is causing the Portal to produce consistent HTTP 500 errors. At this time the Portal is completely unavailable.</p>tag:status.patchman.co,2005:Incident/192501062023-12-14T10:20:06+01:002023-12-14T10:20:06+01:00Routine security maintenance on Licensing cluster<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:20</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:15</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:05</var> CET</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:03</var> CET</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Licensing API offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>14:35</var> CET</small><br><strong>Scheduled</strong> - We will be performing routine maintenance on our Licensing API cluster on Thursday 14 December starting from 10:00 (CEST). The Licensing API will be unavailable for a maximum expected timespan of 5 minutes. During this time, new server registrations may fail and the agents may log warnings. All of these tasks can be safely retried. Existing licenses will remain functional and all other services will remain available and functional.</p>tag:status.patchman.co,2005:Incident/192500942023-12-12T10:29:20+01:002023-12-12T10:29:20+01:00Routine security maintenance on Portal cluster<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:29</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:22</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:18</var> CET</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:15</var> CET</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Portal, Customer API and Data Processing Infrastructure offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>14:33</var> CET</small><br><strong>Scheduled</strong> - In the morning of Tuesday 12 December, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected maximum timespan of 5 minutes.</p>tag:status.patchman.co,2005:Incident/192500842023-12-07T11:56:01+01:002023-12-07T11:56:01+01:00Routine security maintenance on Portal cluster<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:56</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:42</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:36</var> CET</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:32</var> CET</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Portal, Customer API and Data Processing Infrastructure offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>14:32</var> CET</small><br><strong>Scheduled</strong> - In the morning of Thursday 7 December, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected maximum timespan of 5 minutes.</p>tag:status.patchman.co,2005:Incident/187541672023-10-19T10:13:41+02:002023-10-19T10:13:41+02:00Routine security maintenance on Licensing cluster<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:13</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:09</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:06</var> CEST</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:05</var> CEST</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Licensing API offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - The maintenance window has started. We will first be performing various preparatory steps that do not involve downtime for any of the affected services.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>15:19</var> CEST</small><br><strong>Scheduled</strong> - We will be performing routine maintenance on our Licensing API cluster on Thursday 19 October starting from 10:00 (CEST). The Licensing API will be unavailable for a maximum expected timespan of 5 minutes. During this time, new server registrations may fail and the agents may log warnings. All of these tasks can be safely retried. Existing licenses will remain functional and all other services will remain available and functional.</p>tag:status.patchman.co,2005:Incident/187541522023-10-18T10:28:55+02:002023-10-18T10:28:55+02:00Routine security maintenance on Portal cluster<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:28</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:23</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:17</var> CEST</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:15</var> CEST</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Portal, Customer API and Data Processing Infrastructure offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:01</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:01</var> CEST</small><br><strong>Update</strong> - The maintenance window has started. We will first be performing various preparatory steps that do not involve downtime for any of the affected services.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>15:18</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Wednesday 18 October, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected maximum timespan of 5 minutes.</p>tag:status.patchman.co,2005:Incident/187542282023-10-17T10:22:47+02:002023-10-17T10:22:47+02:00Routine security maintenance on Portal cluster<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>10:22</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>10:18</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>10:11</var> CEST</small><br><strong>Update</strong> - All services are operational again. We will continue the final steps of our maintenance procedures without impacting availability.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>10:08</var> CEST</small><br><strong>Update</strong> - As part of our maintenance, we will now perform the procedure that requires taking the Portal, Customer API and Data Processing Infrastructure offline for a maximum of 5 minutes. We will update this incident once the services are back online.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>15:26</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Tuesday 17 October, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected maximum timespan of 5 minutes.</p>tag:status.patchman.co,2005:Incident/183822642023-09-22T10:09:47+02:002023-09-22T10:09:47+02:00Partial outage on the Data Processing Infrastructure.<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>10:09</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:56</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>14:00</var> CEST</small><br><strong>Investigating</strong> - Currently, we are seeing a high volume of detection and task data that is in the backlog for processing. As a result of this, the feedback between the agents and Portal is slow, making it potentially seem unresponsive.</p>tag:status.patchman.co,2005:Incident/183236932023-08-27T05:00:00+02:002023-08-30T19:53:35+02:00Portal backend unreachable<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>05:00</var> CEST</small><br><strong>Resolved</strong> - We're currently investigating problems with the Patchman Portal being unavailable for use due to an outage on the backend.</p>tag:status.patchman.co,2005:Incident/175472772023-06-12T18:13:07+02:002023-06-12T18:13:07+02:00Portal backend unreachable<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>18:13</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>16:17</var> CEST</small><br><strong>Update</strong> - The outage has been resolved, but, we are still completing some cleanup on the backend to ensure it doesn't regress.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>15:21</var> CEST</small><br><strong>Update</strong> - Systems have resumed operation, but we are still remediating the root cause of the issue.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>15:00</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>14:56</var> CEST</small><br><strong>Investigating</strong> - We're currently investigating problems with the Patchman Portal being unavailable for use due to an outage on the backend.</p>tag:status.patchman.co,2005:Incident/120133932022-10-24T10:52:37+02:002022-10-24T10:52:37+02:00Routine security maintenance on Portal cluster<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:52</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:13</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>11:41</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Monday 24th October, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected time span of maximum 5 minutes.</p>tag:status.patchman.co,2005:Incident/118235722022-10-11T13:57:31+02:002022-10-11T13:57:31+02:00Portal web-interface unavailable.<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>13:57</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>12:33</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>12:19</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating a problem with the Portal web-interface being unavailable.</p>tag:status.patchman.co,2005:Incident/113105072022-10-06T10:54:55+02:002022-10-06T10:54:55+02:00Routine security maintenance on Portal cluster<p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>10:54</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>10:20</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:08</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Thursday 6th October, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected time span of maximum 5 minutes.</p>tag:status.patchman.co,2005:Incident/110712232022-09-15T11:04:58+02:002022-09-15T11:04:58+02:00Routine security maintenance on Portal cluster<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>11:04</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>10:30</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>11:42</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Thursday 15th September, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected time span of maximum 10 minutes.</p>tag:status.patchman.co,2005:Incident/110711812022-09-14T10:41:05+02:002022-09-14T10:41:05+02:00Routine security maintenance on Portal clusters<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:41</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:34</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>11:32</var> CEST</small><br><strong>Scheduled</strong> - In the morning of Wednesday 14th September, we will be performing routine security maintenance on our Portal cluster starting from 10:00 (CEST). During this time, the Portal, Customer API and Data Processing Back-ends will be temporarily unavailable for an expected time span of maximum 10 minutes.</p>tag:status.patchman.co,2005:Incident/110091622022-09-06T06:50:45+02:002022-09-06T06:50:45+02:00Routine security maintenance on Licensing cluster<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>06:50</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>06:22</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>06:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>11:19</var> CEST</small><br><strong>Scheduled</strong> - We will be performing routine maintenance on our Licensing API cluster on September 06 starting from 06:00 (CEST).<br /><br />The Licensing API will be unavailable for a maximum time span of 10 minutes. During this time, new server registrations may fail and the agents may log warnings. All of these tasks can be safely retried. Existing licenses will remain functional and all other services will be available.</p>tag:status.patchman.co,2005:Incident/109314162022-08-26T09:31:01+02:002022-08-26T09:31:01+02:00Intermittent connectivity issue<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>09:31</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>02:22</var> CEST</small><br><strong>Monitoring</strong> - Our infrastructure provider was able to correct the issues affecting connectivity and all services are available again. We keep monitoring the situations for regressions.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>01:14</var> CEST</small><br><strong>Update</strong> - Our infrastructure provider has identified the problem and is currently implementing a fix. We will continue to work with them in resolving this problem as soon as possible.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>00:48</var> CEST</small><br><strong>Identified</strong> - Our infrastructure provider is experiencing networking problems, which are leading to intermittent connectivity issues on our platform. All our APIs are experiencing packet loss and/or slow response times. We are working with our provider to resolve the problem as soon as possible.</p>tag:status.patchman.co,2005:Incident/104430412022-07-07T10:51:31+02:002022-07-07T10:51:31+02:00Portal back-end software upgrade<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>10:51</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>10:35</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>14:26</var> CEST</small><br><strong>Scheduled</strong> - We will be performing a software upgrade on the Portal back-end in the morning of July 07 starting from 10:00 (CEST). The goal of this upgrade is to maintain/increase stability of the back-end components. <br /><br />During this maintenance the Agent API will be unavailable for a maximum time span of 30 minutes, resulting in temporary connection loss between the Patchman Agent and our infrastructure. The Portal and Customer API will be unavailable for a maximum time span of 10 minutes.<br /> <br />When the maintenance is completed the connections are automatically restored.</p>tag:status.patchman.co,2005:Incident/102783492022-06-21T15:57:28+02:002022-06-21T15:57:28+02:00Load-balancing cluster migrations<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>15:57</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>12:34</var> CEST</small><br><strong>Update</strong> - The Portal and Customer API are available again and no further downtime is expected on these components.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>12:12</var> CEST</small><br><strong>Update</strong> - We will now start transitioning the Portal Load-balancing environment. This might effect active connections to the Portal web-interface and Customer-API but connectivity should be restored in a few seconds.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>12:08</var> CEST</small><br><strong>Update</strong> - The Agent-API is available again and no further downtime is expected on this component.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>11:51</var> CEST</small><br><strong>Update</strong> - We will now start transitioning the Agent-API Load-balancing environment.<br /><br />This might result in loss of connections between agent installations and our platform, and a slight delay in reporting and execution of detection's, scans and agent-side tasks. All connections will be automatically re-established when the maintenance concludes.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>11:49</var> CEST</small><br><strong>Update</strong> - The Licensing-API is available again and no further downtime is expected on this component.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>11:28</var> CEST</small><br><strong>Update</strong> - We will now start transitioning the Licensing Load-balancing environment. During this transition, new server registrations may fail and the agents may log warnings. <br /> <br />All of these tasks can be safely retried. Existing licenses will remain functional.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>10:27</var> CEST</small><br><strong>Scheduled</strong> - Starting from 10:00 (CEST) on June 21 we will be doing maintenance on our Load-balancing servers. To ensure the stability and reliability of our Load-balancing systems we will be transitioning to another solution on all clusters. <br /><br />While we don't expect significant downtime, a brief moment of lost connectivity might occur while we are transitioning between solutions. <br /><br />During these transitions we will update this maintenance announcement on a per cluster basis.</p>tag:status.patchman.co,2005:Incident/102540352022-06-13T08:30:13+02:002022-06-13T08:30:13+02:00Degraded performance on data processing infrastructure<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>08:30</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>00:25</var> CEST</small><br><strong>Monitoring</strong> - Performance is back to normal. We will continue to monitor the situation to ensure no further regressions.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>00:21</var> CEST</small><br><strong>Update</strong> - Portal and Customer API are no longer affected, remediating actions for the backend workers are still ongoing.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>00:15</var> CEST</small><br><strong>Update</strong> - As part of the remediation steps, there may be a temporary loss of availability of the Portal and Customer API, expected to be limited to a maximum of 1 minute.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>00:09</var> CEST</small><br><strong>Identified</strong> - Our backend workers are suffering from degraded performance, which we have traced to our database cluster. We are currently preparing actions to redirect the load to other cluster workers as a means of remediating the performance problems.</p>tag:status.patchman.co,2005:Incident/100818092022-05-26T20:31:44+02:002022-05-26T20:31:45+02:00Errors on Portal, Customer API, Agent API<p><small>May <var data-var='date'>26</var>, <var data-var='time'>20:31</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>19:30</var> CEST</small><br><strong>Monitoring</strong> - All issues are resolved and we are monitoring the performance of these components.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>19:22</var> CEST</small><br><strong>Identified</strong> - We are seeing a major outage across several components, resulting in high error rates. The cause has been identified, and we are working on resolving the problem.</p>tag:status.patchman.co,2005:Incident/97617382022-04-12T00:07:05+02:002022-04-12T00:07:05+02:00New connections by Agent software to services rejected<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>00:07</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>21:44</var> CEST</small><br><strong>Monitoring</strong> - The issue has been resolved, and all connections are restoring.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>21:30</var> CEST</small><br><strong>Identified</strong> - We are currently seeing rejection of automated connections initiated by the Agent software to our backend systems, particularly the Definitions API. We have identified the root cause and are working to resolve this. Aside from being unable to update definitions, customers may see that single sign-on for end users from their customer panels at this time does not work.<br /><br />Any connections currently rejected will automatically restore as soon as the problem is resolved.</p>tag:status.patchman.co,2005:Incident/95656652022-03-24T06:38:58+01:002022-03-24T06:38:58+01:00Routine security maintenance on Licensing cluster<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>06:38</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>06:22</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>06:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>14:37</var> CET</small><br><strong>Scheduled</strong> - We will be performing routine maintenance on our Licensing API cluster on March 24 starting from 06:00 (CET).<br /><br />The Licensing API will be unavailable for a maximum time span of 5 minutes. During this time, new server registrations may fail and the agents may log warnings. All of these tasks can be safely retried. Existing licenses will remain functional and all other services will be available.</p>