Back to all articlesIntegration reliability

Decide when a dependency incident is resolved and restrictions can be removed

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

The provider has marked its incident resolved, but your integration still returns occasional errors. Some work remains unfinished, and the restrictions that protected the system are still active. Removing everything could recreate the failure; leaving everything restricted could prolong the disruption.

Accept recovery using evidence from the affected business workflow, not just the dependency. Separate permission to test more traffic from permission to remove controls and close the incident. Each decision needs observable criteria and someone authorised to stop restoration.

The filled hypothetical checklist below shows how to make those decisions without treating limited-load success as demonstrated recovery. Its thresholds are illustrative, not recommended defaults.

1. Separate provider recovery from operational closure

A provider notice describes the provider’s assessment. It does not establish whether your credentials work, affected operations complete, or staff can safely resume normal procedures.

Use distinct states:

  • Provider reports recovery: an external signal has changed; local verification begins.
  • Integration recovery verified: representative operations complete correctly under the tested conditions.
  • Normal traffic demonstrated: the intended workload has been observed within agreed operating bounds.
  • Operationally closed: residual work has an accepted handover, incident controls have explicit dispositions, and staff know what to do.

These states need not change together. An interactive workflow may recover before bulk processing. An incident may close with known exceptions if ordinary operations can manage them without continued incident coordination. State the boundary rather than calling every partial recovery complete.

2. Define evidence before relaxing restrictions

Write the acceptance rule while restrictions remain active. Otherwise, a brief improvement can become an excuse to lower the standard after observing the results.

For each affected workflow, specify its scope, expected business outcome, health bounds, observation duration, workload coverage, evidence location, and decision authority. Use normal operating expectations and failure consequences to select those criteria.

Separate read and write evidence. A successful read does not prove a previously failing write works. An acknowledgement does not prove an update finished. Where writes were uncertain, reconcile destination records and check for missing or duplicate effects.

Define what must be collected again after each restoration step. Earlier evidence remains useful history, but it cannot establish performance under a newly increased workload.

Unknown is not pass. Missing telemetry, untested operations, and unreconciled results should block acceptance for that scope or produce an explicitly narrower, provisional decision.

3. Use this filled recovery acceptance record

Hypothetical worked example: A service business uses a CRM integration to exchange job information with an external scheduling provider. Slow acknowledgements and uncertain writes led the team to limit interactive updates, pause bulk updates, and keep a manual exception log.

All people, identifiers, times, quantities, thresholds, and outcomes below are fictional. These are recommended operating arrangements, not claims about Alactic controls or capabilities. Evidence identifiers are example record labels, not links.

Record and authority

  • Incident: DEP-EXAMPLE; scope covers job reads, interactive updates, and bulk updates.
  • Provider notice: recovery reported at 09:10; supporting context only.
  • Recovery lead: Maya, integration operations; authorised to approve restoration stages.
  • Business owner: Leon, service operations; confirms usable job information and accepts residual work.
  • Rollback operator: Priya, on-call engineer; present during restoration and authorised to reapply restrictions.

Gate A: observed integration health

  • Rule at every stage: record reads and writes separately; no timeouts, unexplained outcomes, missing updates, or duplicate effects. Each observed request must complete within the example’s five-second limit. Reconcile every tested write against its intended destination result.
  • Initial check, 09:30: HOLD. Two diagnostic writes remained unconfirmed. Both were later located and verified; diagnostic tests are recorded separately from the incident-period inventory below.
  • Restricted stage, 10:00–10:30: 30 reads returned expected records; 60 writes completed and reconciled. Both previously failing update types were covered. No rule breaches. Evidence: A-READ-1, A-WRITE-1, and A-RECON-1.
  • Decision, 10:35: PASS for testing normal interactive traffic only. Bulk processing remains paused.
  • Repeat requirement: every later stage needs fresh read counts and results, write counts and reconciliation, timing and timeout records, workload coverage, and a signed decision. Gate D records that evidence.

Gate B: incident-period task accounting

  • Inventory: 240 incident-period updates, excluding the separately logged diagnostic and restoration tests.
  • Reconciliation: 205 confirmed complete; 20 cancelled because newer information superseded them; 15 have confirmed destination outcomes but require a business decision about the next action.
  • Meaning of the 15 exceptions: the intended updates did not take effect, and reconciliation found no duplicate effect. Whether to apply revised information, cancel, or contact staff remains undecided.
  • Evidence: B-LEDGER-1 matches all 240 source items to destination findings, cancellation reasons, or exception records.
  • Decision: PASS for technical accounting. No technical outcomes remain unknown; business disposition for 15 items remains pending. This is not a claim that all work is complete.

Gate C: residual-work handover

  • Control: the 15 pending business exceptions remain excluded from automatic resubmission.
  • Accepted owner: Leon accepts the list at 11:00 and assigns each item to a named service-operations colleague in C-EXCEPTIONS-1.
  • Checkpoint: 15:00 the same day. Each item must show a completed disposition or a documented next action, owner, and next checkpoint.
  • Escalation: an exception blocking a current service job returns to active operational escalation immediately.
  • Decision: PASS for handover, not for completion of the pending business decisions.

Gate D: temporary-control removal

  • Interactive limit: relaxed at 10:40. From 10:40–11:10, 40 reads returned expected records and 80 writes completed and reconciled. No health-rule breaches. Evidence: D-INTERACTIVE-READ, D-INTERACTIVE-WRITE, and D-INTERACTIVE-RECON.
  • Interactive comparability: the recorded reference range for comparable normal half-hour periods was 35–45 reads and 70–90 writes, covering the same operation types. Maya accepts interactive restoration at 11:10.
  • Bulk pause: removed for one authorised scheduled batch at 11:15. From 11:15–12:00, 300 bulk writes completed; reconciliation confirmed all 300 intended destination changes without duplicates or unknown outcomes. During that interval, 60 reads returned expected records and 90 interactive writes completed and reconciled. No health-rule breaches.
  • Bulk comparability: the intended normal batch range was 250–350 updates within a 45-minute observation period, alongside interactive demand. The observed batch covered both previously failing update types. Evidence: D-BATCH-MANIFEST, D-BULK-RECON, D-CONCURRENT-READ-WRITE, and D-TIMING.
  • Manual log: new incident entries stopped at 12:05 after reconciliation; existing records remain available. The separate exception list remains active.
  • Decision, 12:10: PASS for the documented workload envelope. Larger or materially different workloads require separate assessment. Any timeout, unexplained result, or timing breach stops the next change and triggers assessment of restriction reinstatement.

Gate E: communication and closure

  • Staff message: normal job-update procedures may resume within the accepted workload. Do not resubmit items on the incident exception list.
  • Remaining work: 15 technically reconciled exceptions still await business disposition under Leon’s ownership; the checkpoint remains 15:00.
  • Receipt: service-desk and scheduling leads acknowledge the instructions and relay them to their shifts.
  • Reopening route: send any new uncertain outcome to the on-call channel with its task identifier.
  • Decision, 12:15: Maya and Leon accept operational closure, linking the evidence and exception handover. The decision does not mark the 15 exceptions complete.

Gate F: measurable learning action

  • Observed gap: early checks focused on acknowledgements instead of completed destination changes.
  • Action: Priya owns a proposed outcome-reconciliation check, due for review within five working days.
  • Completion test: a controlled exercise must identify deliberately incomplete and duplicate outcomes and assign them before recovery acceptance.
  • Uncertainty retained: the provider’s underlying failure explanation remains pending; it is not substituted for local evidence.

4. Treat low-load success as permission for the next test

Restrictions can make an integration look healthy precisely because they suppress demand. In the example, the first passing stage justified an interactive test, not unrestricted bulk processing.

Compare observed demand with the workload you intend to admit. Include operation mix, concurrency, batch size, and elapsed observation time where relevant. A completed small batch does not establish that the ordinary scheduled batch is safe.

If representative demand never arrives, record provisional restoration instead of a full pass. For example, a hypothetical decision could retain the bulk cap overnight, assign Maya to observe the next scheduled batch, and withhold bulk acceptance until that batch is reconciled. Give the checkpoint a specific time and evidence requirement.

Change one meaningful control at a time where practical. This makes the effects easier to interpret and reduces uncertainty about which change exposed a renewed failure.

5. Retain useful controls without preserving unsafe workarounds

For each incident workaround, record its purpose, removal prerequisite, owner, reversal method, and post-removal check. Stopping a manual process does not require deleting its evidence.

Some controls should remain. The exception resubmission block in the example protects items awaiting business decisions; closing the incident does not justify removing it. Give retained controls an operating impact and review date so they do not become unexplained permanent restrictions.

Do not treat the pre-incident configuration as automatically safe. If restoring it would recreate a known risk, agree a new operating baseline. Likewise, ordinary security and business safeguards are not incident debris to remove during cleanup.

6. Distinguish known outcomes from finished business decisions

Technical certainty and business completion answer different questions. Knowing that an update did not occur resolves uncertainty about its effect. It does not decide whether the business still wants that update performed.

This distinction allows an honest handover without keeping every incident open indefinitely. The example’s 15 items can move to service operations because their technical outcomes are known, their scope is explicit, and an owner has accepted the pending decisions.

Keep incident coordination active if unresolved uncertainty threatens current operations or requires multiple teams to intervene together. A small count of ambiguous writes can justify more attention than a large list of clearly cancelled tasks. Quantity alone is a poor closure gate.

7. Reopen the affected gate when recovery fails

Assign an observer during each restoration stage who can stop the next change. If a gate fails:

  1. Pause further restoration and identify the affected workflow.
  2. Assess whether the accepted operating assumptions still hold.
  3. Reapply the last acceptable restriction where appropriate, rather than repeatedly toggling controls.
  4. Account for work created during the failed stage before allowing resubmission.
  5. Collect fresh evidence for the affected gate and communicate renewed staff restrictions.

A later failure does not erase earlier observations. It invalidates acceptance claims whose assumptions no longer hold. Preserve both decisions so the next shift can understand what changed.

8. Make learning testable after operational closure

Restoration approval and improvement delivery are separate decisions. The service may be usable before a monitoring or reconciliation improvement is implemented.

Replace vague actions such as improving monitoring with a missing signal and a demonstration. In the example, completion requires detecting deliberately incomplete and duplicate outcomes, not merely adding a dashboard.

When reviewing that exercise, compare the recorded acceptance decision with the injected faults. If a fault escaped detection, revise the gate or evidence collection before considering the action complete. This turns the incident record into a testable improvement rather than another checklist filed away.

Integration reliabilityIncident recoveryOperational readiness

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications