During my period at ING Bank, I was part of the forward-thinking 4WRD DevOps team, focusing on the Basel II reporting system. This project involved transitioning a mainframe application to the ING Data Lake, utilizing Netezza Data Warehouse and IBM Infosphere Datastage for improved data management and reporting.
There where two major challenges:
- Understanding the inner workings of the legacy system that was for a large part based on mainframe DB2 data and COBOL code.
- Creating a new working solution based on Datastage and Netezza. Both tools nobody in the team had worked with before
At first sight this looked overwelming for our small team of six. We took a practical approach of learning on the job. We started with the easiest looking datastream and implement a full end to end data flow. We learned and gradually improved our methods for documenting, coding and testing.
My role was pivotal in establishing a robust deployment pipeline, crafting ETL (Extract, Transform, Load) data transformations, executing monthly Basel reporting batches, and offering comprehensive technical support to our team. In overseeing operations, I ensured the system’s reliability and efficiency. Moreover, as a developer of team tools, I innovated automated deployment strategies using a suite of technologies including Gitlab, Jenkins, Artifactory, Ansible, and Bash scripting, coupled with IBM Datastage for deployment tool sets.
Our team’s accomplishments are notable. We successfully processed monthly data for over 8 million customers regarding unsecured loans and credit cards, using IBM Datastage ETL jobs to handle the data and storing the outcomes in a Netezza warehouse. We pioneered a continuous deployment process, integrated FitnesseDB for regression testing, and meticulously stored every code asset on Gitlab. This comprehensive approach not only facilitated seamless deployment but also provided a solid framework for recovery from any setbacks.
In an effort to share our journey and methodologies with others, I created an engaging animation depicting Team 4WRD’s development of the Basel II reporting system. This animation showcased our dynamic workflow and updates to various assets, produced using the open-source visualization tool Gource.
Tauvic's Blog 