tag:status.myrecoverylink.com,2005:/historyRecoveryLink Status - Incident History2024-03-29T05:22:57ZRecoveryLinktag:status.myrecoverylink.com,2005:Incident/191310742023-11-15T23:25:32Z2023-11-15T23:25:32ZRecoveryLink ERR Intermittent Login Issues<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>23:25</var> UTC</small><br><strong>Resolved</strong> - RecoveryLink engineers have identified an issue with the cloud hosting provider that is affecting certain authentication processes for user accounts. The team will be monitoring the updates being made by the provider to resolve the reported issue.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>21:33</var> UTC</small><br><strong>Investigating</strong> - Our engineers are currently investigating reports of intermittent login issues by some RecoveryLink application users.</p>tag:status.myrecoverylink.com,2005:Incident/187440992023-10-10T17:01:01Z2023-10-10T17:01:01ZRecoveryLink ERR Application Login Issues<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. RecoveryLink engineers will continue to monitor updates for any status changes.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>Monitoring</strong> - RecoveryLink engineers are monitoring issue updates. ERR application login corrected in testing.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>14:44</var> UTC</small><br><strong>Update</strong> - Our engineers are seeing partial resolution to login function and are still investigating initial cause for any service disruption.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Investigating</strong> - Our engineers are currently investigating reports of general login issues for the ERR application. Updates will be posted throughout our review.</p>tag:status.myrecoverylink.com,2005:Incident/183209242023-08-31T09:10:15Z2023-08-31T09:10:15ZParticipant Profile Field Propagation Issue in Default Engagement Session Templates<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>09:10</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>18:45</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>13:35</var> UTC</small><br><strong>Investigating</strong> - We have confirmed a global bug introduced in last nights release to fix participant context that is preventing the pre-population of blocks with "update revise" behavior set. Our team is working quickly to identify and resolve this issue. In the interim, we would suggest removing the participant profile blocks (contact info, demographics, recovery characteristics) from your session templates to prevent two things: 1) your staff from having to fill them out again to submit sessions, and 2) to prevent over writing of previously saved data. The enterprise admin for each location can remove these blocks from the brief check-in, recovery planning, and care coordination session templates via the customization engine, or can request assistance by emailing our support team to perform the short term change on their behalf.<br /><br />Please note that the recovery plan block is also impacted by this behavior so we would encourage customers to not submit recovery planning sessions until the issue is resolved.</p>tag:status.myrecoverylink.com,2005:Incident/181841112023-08-19T13:27:00Z2023-08-19T13:27:00ZCore ERR Platform: Issues in Setting Participant Context When Viewing Profile<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>13:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>13:47</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:41</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue in which the core ERR application fails to correctly set a participant in context once selecting a profile. For a temporary solution, please select the participant and then refresh your browser before navigating to sections of the participant profile. In some cases, navigation in the participant profile side bar may also be impacted. Users may first click on a menu item (e.g., engagements) before attempting to navigate their cursor to the submenu to improve functionality.</p>tag:status.myrecoverylink.com,2005:Incident/179123822023-07-20T15:28:54Z2023-07-20T15:28:54ZIntake Submit Failure Global Impact<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>15:28</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>15:05</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:07</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a global platform issue that is preventing new intakes from submitting successfully. Other content types (engagements, custom forms, and activity logs) remain operational.</p>tag:status.myrecoverylink.com,2005:Incident/175830182023-06-15T19:37:00Z2023-06-15T19:37:00ZRecoveryLink 2.0 - Data Saving Issue<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>15:23</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>13:42</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a critical issue that is preventing content from being saved in RecoveryLink 2.0 (legacy is not impacted at this time).</p>tag:status.myrecoverylink.com,2005:Incident/173436012023-05-23T21:41:44Z2023-05-23T21:42:28ZDegraded Search Functions (RecoveryLink 1.0)<p><small>May <var data-var='date'>23</var>, <var data-var='time'>21:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Investigating</strong> - We are monitoring a reported provider issue with Apache Solr service degradation which is impacting participant search functions in legacy RecoveryLink 1.0. And update has been deployed and development team is monitoring the incident.</p>tag:status.myrecoverylink.com,2005:Incident/164937542023-03-14T13:24:41Z2023-03-14T13:24:41ZLegacy Application - Login Issues<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>13:24</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>13:20</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>13:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.myrecoverylink.com,2005:Incident/107988012022-08-10T22:29:08Z2022-08-10T22:29:14ZRecoveryLink ERR application outage August 10, 2022<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>22:29</var> UTC</small><br><strong>Resolved</strong> - This incident is being closed. RecoveryLink developers are continuing to work with cloud providers on evaluating impacted services.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>18:04</var> UTC</small><br><strong>Monitoring</strong> - Service has been restored to the RecoveryLink ERR application and developers are monitoring cause of incident with cloud service providers.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a global issue with RecoveryLink ERR application servers.</p>tag:status.myrecoverylink.com,2005:Incident/107541562022-08-04T14:09:34Z2022-08-04T14:09:34ZRecoveryLink ERR application outage August 3, 2022<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>14:09</var> UTC</small><br><strong>Resolved</strong> - Updates completed and support team is monitoring application performance.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>14:08</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>Update</strong> - Our development team will be implementing and testing an update for the RecoveryLink ERR application this evening. We will remain in monitoring status until this process this complete.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>17:58</var> UTC</small><br><strong>Update</strong> - System restoration updates are being monitored.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>17:57</var> UTC</small><br><strong>Monitoring</strong> - Service has been restored to the primary application. Our team is monitoring the update and verifying all operating components.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>17:41</var> UTC</small><br><strong>Investigating</strong> - We are experiencing a global service outage to the primary RecoveryLink ERR application and related services. Support team is currently investigating cause with hosting cloud provider. This incident will be updated as soon as the outage cause is identified.</p>tag:status.myrecoverylink.com,2005:Incident/99837912022-05-13T14:45:31Z2022-05-13T14:45:31ZElectronic Recovery Record Record Saving Issue<p><small>May <var data-var='date'>13</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Resolved</strong> - We have resolved an issue that is preventing saving of several types of records in the electronic recovery record platform. This update will continue to be monitored.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>14:34</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:35</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that is preventing saving of several types of records in the electronic recovery record platform.</p>tag:status.myrecoverylink.com,2005:Incident/99051552022-05-04T03:50:49Z2022-05-04T03:50:49ZDocument Signing Feature Maintenance Operation<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>03:50</var> UTC</small><br><strong>Completed</strong> - Maintenance operations for the Document Signing Feature have been completed and are being monitored.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>22:05</var> UTC</small><br><strong>Verifying</strong> - We are applying a series of updates to the Document Signing Feature which will extend the maintenance operation.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>16:36</var> UTC</small><br><strong>Scheduled</strong> - We are undergoing a maintenance operation for the Document Signing feature at this time.</p>tag:status.myrecoverylink.com,2005:Incident/98358192022-04-22T18:53:43Z2022-04-22T18:53:43ZDegraded performance for RecoveryLink core ERR service<p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>18:53</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>18:50</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Investigating</strong> - Earlier today, we were experiencing intermittent issues with one of our cloud providers who have been working to ensure the issue has been resolved. Status updated to monitoring by RecoveryLink developers.</p>tag:status.myrecoverylink.com,2005:Incident/98055402022-04-18T17:46:43Z2022-04-18T17:46:43ZDisplay of participants on active / search navigation pages<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:36</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:18</var> UTC</small><br><strong>Identified</strong> - We have identified the likely cause of the issue and are working with our cloud services provider to confirm and implement a fix.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Investigating</strong> - We are aware of a display issue impacting the availability of participants on the Active and Search participant tabs. We are currently investigating the issue and will provide more information as it becomes available.</p>tag:status.myrecoverylink.com,2005:Incident/88139682021-12-10T23:09:15Z2021-12-10T23:09:15ZTelerecovery Service Provider Issues<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>23:09</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>22:22</var> UTC</small><br><strong>Update</strong> - Service provider reports operational status for Telerecovery. The Product Team will continue to monitor for resolution.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>20:59</var> UTC</small><br><strong>Monitoring</strong> - We are currently monitoring a reported issue by RecoveryLink Telerecovery service provider, Twillio. As of 12:51 PST, Twillio is investigating an issue with latency and timeout errors on their Rest API. This is currently impacting RecoveryLink SMS services. <br /><br />Our Product Team is monitoring provider reporting.</p>tag:status.myrecoverylink.com,2005:Incident/86182062021-11-20T05:49:28Z2021-11-20T05:49:28ZRecoveryLink Telerecovery Scheduled Services Update 2021-11-19<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>05:49</var> UTC</small><br><strong>Completed</strong> - The scheduled update for RecoveryLink telerecovery services has been completed.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>23:58</var> UTC</small><br><strong>Scheduled</strong> - A maintenance window is being set for RecoveryLink Telerecovery services to implement a script required after a recent product release. Telerecovery services may be interrupted Friday, November 19, between the hours of 11:00 pm CST and Midnight.</p>tag:status.myrecoverylink.com,2005:Incident/85451522021-11-18T03:14:04Z2021-11-18T03:14:04ZRecoveryLink Database Update<p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>03:14</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>03:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>20:36</var> UTC</small><br><strong>Scheduled</strong> - RecoveryLink engineers will be upgrading the memory allocation for core databases. The upgrade requires database servers to be restarted which will restrict dashboard analytics access during the scheduled maintenance window - Wednesday, November 17 at 10:00 pm EST.</p>tag:status.myrecoverylink.com,2005:Incident/83405352021-10-28T16:03:04Z2021-10-28T16:03:04ZTelerecovery Availability<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>15:38</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.myrecoverylink.com,2005:Incident/81534122021-10-05T00:58:53Z2021-10-05T00:58:53ZDatabase ETL updates 2021-10-04<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>00:58</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>00:22</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>00:12</var> UTC</small><br><strong>Scheduled</strong> - Update to RecoveryLink ETL system is being completed to manage recent data model changes in advance of Q3 reporting. Potential disruption to dashboard analytics may occur for a 30-minute period.</p>tag:status.myrecoverylink.com,2005:Incident/81209772021-09-30T13:28:47Z2021-09-30T13:28:47ZElectronic Recovery Record Data Records Visibility<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>13:28</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>13:15</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>12:55</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue where previously collected data is not consistently visible within certain individual session types within the electronic recovery record.</p>tag:status.myrecoverylink.com,2005:Incident/80032862021-09-15T22:19:03Z2021-09-15T22:19:03ZDatabase ETL updates 2021-09-15<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>22:19</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>22:04</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>21:37</var> UTC</small><br><strong>Scheduled</strong> - Update to RecoveryLink ETL system is being completed to manage recent data model changes in advance of mid-month reporting. Potential disruption to dashboard analytics may occur for a 30-minute period.</p>tag:status.myrecoverylink.com,2005:Incident/74571392021-07-09T18:21:19Z2021-07-09T18:21:19ZMaintenance Update for Enterprise Dashboard 2021-07-09<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>18:21</var> UTC</small><br><strong>Resolved</strong> - Maintenance is complete for Enterprise Dashboard and it is operational.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>17:02</var> UTC</small><br><strong>Identified</strong> - We are currently completing an update protocol for the Enterprise Dashboard based on incident from 2021-07-08.</p>tag:status.myrecoverylink.com,2005:Incident/74381012021-07-08T01:13:05Z2021-07-08T01:13:05ZEnterprise dashboard login issue<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>01:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved and the team will continue monitoring the updates applied for secure operations. An additional protocol has been added to the database ETL update process to protect against changes to references tables that inform the underlying login and permissions controls.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>23:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and is being tested in development environment before release to production for customer access.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>22:48</var> UTC</small><br><strong>Identified</strong> - We have identified the source of the problem with the ETL to a reference database table. Applying updates to resolve.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>21:31</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue associated with login to the Enterprise Dashboard. Operations have ben suspended for maintenance work.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>21:04</var> UTC</small><br><strong>Investigating</strong> - The Enterprise Dashboard is being taken offline while we investigate an issue with login access.</p>tag:status.myrecoverylink.com,2005:Incident/74358492021-07-07T16:46:44Z2021-07-07T16:46:44ZEnterprise dashboard Historic Data error<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved with a reinstallation of R package to production server.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>15:44</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue with an R package update (NavPillsItem) for shinydashboardPlus on the production server. This is currently impacting reporting of Historic Data.</p>tag:status.myrecoverylink.com,2005:Incident/72654032021-06-16T19:59:38Z2021-06-16T19:59:38ZPartner Site (Pantheon) Experiencing Infrastructure Issues<p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>19:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>19:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>19:05</var> UTC</small><br><strong>Investigating</strong> - We are currently working with our partner to investigate the issue.</p>