Build incident response for scalable data pipelines with defined severity and escalation.
Building Scalable Data Processing Pipelines
Move scalable data pipelines from general awareness to a repeatable, reviewable practice.
📍 Cairo🗓️ 5 training days📚 4 modules🎓 Accredited certificate
5intensive training days
4scientific modules
8training sessions
32detailed points
Course Overview
The documentation for scalable data pipelines is accurate only until the next release. Availability targets for the practice within technology and systems delivery are commercial commitments, whatever the engineering team calls them. Participants apply this aspect of technology and systems delivery to their own technical platform throughout, so the output is directly usable. Applied research in technology and systems delivery consistently shows that early structure around scalable data pipelines reduces downstream rework. Participants take away a working set of documents supporting the wider technology and systems delivery agenda, ready to be adapted internally. The version of the technology and systems delivery discipline described in the manual and the version practised on the floor tend to diverge over time. Participants come from operational and oversight roles, and both perspectives on scalable data pipelines are used deliberately. The course sets out a working method for this area of technology and systems delivery that participants can apply the week they return. The closing exercise tests whether the participant's plan for the practice within technology and systems delivery survives a hostile question.
Expected Learning Outcomes
Build the internal capability for scalable data pipelines rather than depending on external support indefinitely.
Plan the migration or decommissioning of legacy components in scalable data pipelines.
Design a training and briefing approach that sustains competence in scalable data pipelines.
Design a practical operating method for scalable data pipelines that fits the organisation's size and maturity.
Assess and manage third-party and cloud dependencies in scalable data pipelines.
Document scalable data pipelines to the level a new engineer could operate it.
Who Should Attend
Specialists advising senior management on scalable data pipelines.
IT managers and service owners responsible for scalable data pipelines.
Quality and test engineers verifying scalable data pipelines.
Internal auditors reviewing how scalable data pipelines is designed and operated.
IT governance and audit staff reviewing scalable data pipelines.
Service desk and support leads handling incidents in scalable data pipelines.
Course Modules
Scalable data pipelines: build, pipeline and release discipline
2 sessions · 8 pointsSession 1The cost of scalable data pipelines and how to present it
- Run a load test on scalable data pipelines at expected peak plus a margin.
- Document the runbook for scalable data pipelines to the level a new engineer could use.
- Inventory third-party dependencies inside scalable data pipelines and their update status.
- Compare the cost of scalable data pipelines with the cost of its absence.
Session 2Retiring the legacy part of scalable data pipelines
- Write down the assumptions underpinning the approach to scalable data pipelines.
- Confirm that contractual obligations around scalable data pipelines are understood.
- Set out how exceptions to scalable data pipelines are requested and approved.
- State the availability and recovery objectives for scalable data pipelines as numbers.
Scalable data pipelines: incident response and severity
2 sessions · 8 pointsSession 1Setting availability and recovery targets for scalable data pipelines honestly
- Assess the exit route from any cloud or vendor dependency in scalable data pipelines.
- Agree the indicators that will show whether scalable data pipelines is improving.
- Confirm every release of scalable data pipelines can be rolled back within a defined time.
- Record the technical debt in scalable data pipelines and schedule repayment.
Session 2Making releases of scalable data pipelines routine instead of risky
- Anticipate the objections scalable data pipelines will raise and prepare the answers.
- Define incident severity levels for scalable data pipelines and the response each triggers.
- Establish who is informed, consulted and accountable in scalable data pipelines.
- Estimate the resource scalable data pipelines requires to run as designed.
Scalable data pipelines: backup, recovery and continuity
2 sessions · 8 pointsSession 1Monitoring scalable data pipelines from the user's point of view
- Define the trigger that would require scalable data pipelines to be redesigned.
- Decide what will be stopped to create capacity for scalable data pipelines.
- Plan the decommissioning of the legacy component scalable data pipelines replaces.
- Set delivery and reliability indicators for scalable data pipelines the team trusts.
Session 2The hard cases in scalable data pipelines and how to reason about them
- Identify where judgement in scalable data pipelines is legitimate and where it is not.
- Confirm the support model and escalation path for scalable data pipelines.
- Review logging on scalable data pipelines for coverage and retention.
- Plan the sequence in which improvements to scalable data pipelines will be introduced.
Scalable data pipelines: architecture and designing for failure
2 sessions · 8 pointsSession 1Closing out scalable data pipelines and capturing what was learned
- Identify the data already collected that bears on scalable data pipelines.
- Measure current load on scalable data pipelines and project it forward twelve months.
- Distinguish symptoms from causes when scalable data pipelines underperforms.
- Set out the decisions in scalable data pipelines that require sign-off and by whom.
Session 2Documenting scalable data pipelines so someone else can operate it
- Remove steps in scalable data pipelines that add effort without adding assurance.
- Configure alerting on scalable data pipelines that reflects what users experience.
- Review access rights on scalable data pipelines and remove what is no longer needed.
- Apply change control to scalable data pipelines including emergency changes.
Choose the package that suits you
Silver Package
At least 3 people
USD1,250
- Workshop or Program Participation
- Airport Transfers
- Customized Badge
- Expert Mentorship (Private Sessions)
- Supervision & Secretarial Services
- Accredited Certificate of Participation
- Complete Training Kit
- Coffee Break
- Closing Ceremony
Gold Package
At least 3 people
USD1,850
- 5-night stay in a 5-star hotel
- Workshop or Program Participation
- Airport Transfers
- Customized Badge
- Expert Mentorship (Private Sessions)
- Supervision & Secretarial Services
- Accredited Certificate of Participation
- Complete Training Kit
- Coffee Break
- Closing Ceremony
Complete your registration
We will contact you within one business day to confirm.