In my previous blog post i wrote about the SkillMatch system that i am buidling for scanning job portal sites and extracting valuable information. In this post I write about its user interface. Continue reading
How can we extract valuable information from unstructured data such as text documents. Financial compliancy, knowledge management and crime fighting are applications waiting for solutions to mine this data. In this article i describe how i build Skillmatch a job matching system as a practical example how to use this technology. Continue reading
Effective monitoring of thousands of IT Services requires you to transform realtime data streams into actionable alerts and high level situation overviews. In this blog post I will describe how to create reliable alerts and reduce the problem of alert storms by clustering.
This blog is part of a series of blogs on setting up a Smart monitoring system.
In my previous post on monitoring I wrote about setting up customer activity monitoring supported by anomaly detection. Now I will show how to monitor hundreds of services without drowning in data. The solution i present processes raw monitoring metrics and turns that data into actionable alerts and usefull insights.
In this article I’ll describe how I implemented customer activity monitoring and anomaly detection. If you are a service provider that provide services to a group of large accounts its vital to know that your customers can do their business.
Monitoring customer behavior is not only required for managing IT operations its also vital to know from a business point of view. Nowadays customers get smart too. High tech customers such as Netflix and Airbnb use data analytics to monitor the results they get from their payment providers. Based on this data they make real-time decisions to switch to another supplier when that better suits their needs.
My personal experiences programming the Makeblock mBot robot. I’m investigating this robot in order to use it as a tool for teaching technology and programming at an elementary school.
The mBot robot can be programmed with the Makeblock editor an application based on Scratch from MIT Media Lab. The editor and example programs can be downloaded from http://www.mblock.cc/. There are also other options such as on online web browser application and apps for Android and the iPad. Continue reading
Last week I attended the ING Business Continuity conference in Amsterdam as a speaker. More than 140 participants from all over the world discussed for two days how reliability of IT services could be improved, handle major incidents and recover from disasters. My presentation on Smart monitoring provides an practical approach how to improve reliability of IT services.
Hoe kunnen robots worden ingezet op school. De school van mijn dochter maakt dit jaar gebruik van robots zoals de Ozobot als hulpmiddel voor de introductie van logisch denken en programmeren. Als vader en IT-er ben ik geïnteresseerd in de mogelijkheden van deze robot. Daarom kocht ik gelijk het model EVO om eens te kijken wat deze robot wel allemaal kan. Ambitieus als ik ben wilde ik gelijk het hoogst bereikbare onderzoeken en stelde mij als doel: “De robot programmeren als een echte verkeersdeelnemer”. Continue reading
Successful enterprises constantly seek new ways to improve availability of their services, and try to avoid compliance breaches. This can be achieved by managing services on an end-to-end basis. By analyzing the topology of end-to-end business value chains you gain insight into the behavior of your systems in a way you have never seen before.
This weekend all over the world hundreds of servers went down, millions of transactions failed and data was lost due to daylight saving time confusion. This recurring event happens every time we change the clock between winter / summer time because of daylight saving time. In this article I explain the cause of the problem. I also provide some hints on how to prevent these kind of problems. Continue reading
While most companies manage their services on the physical component level some are beginning to manage them at the transaction and business process level. This because they want to understand their customer journey or need to have a complete audit trail in order to meet compliance requirements. In this post i will describe the basics of analyzing business transactions. Continue reading
Microservices promise to deliver new business services faster and at lower cost, but it comes at a price: increased operational complexity. I share my vision on managing microservices on the enterprise level:
- Why you should manage your services
- What challenges you will face
- What benefits you can get
- What you can do to get in control
This article is part of a series on service monitoring.
If you want to offer your customers IT services and promise them 99,9% availability then Service monitoring is essential. Smart Monitoring is a philosophy and way of working to achieve that goal. It enables you to understand your complex IT and be in control. In this article I will give you some guidelines and a basic plan how to get started.
This article is part of a series on monitoring.
The dashboard shows information on several services and how they conform to SLA defined KPI’s such as response time target. You can see how these services perform, what results they return (success, functional and technical errors) and drill down by Team.
I worked for ING Bank in the 4WRD DevOps team for about 3 years on the Basel II reporting system. The mainframe application was moved to the new ING Datalake environment with Netezza Data Warehouse and IBM Infosphere Datastage.
My contribution was developing ETL data transformation jobs, running the monthly BASEL reporting batch in the OPS role and provide technical support for the whole team. In the OPS role I was responsible for running the system. As a team tool developper i developed automated deployment procedures with Gitlab, Jenkins, Artifactory, Ansible en Bash script.
The team achievements:
- Monthly processed data on unsecured loans and creditcards for 8 million customers
- We processed the data with IBM Datastage ETL jobs
- Store the results in a Netezza warehouse.
- We developed a continuous deployment procedure.
- We use FitnesseDB for regression testing
- Every asset in the project is stored as a separate file in Gitlab.
- Datastage Jobs, Sequences, Parameter sets
- Database definition scripts
- Test scripts
- This enables us to have a firm grip on deployment
To show other teams about our work i created the following animation of Team 4WRD developing the Basel II reporting system. You will see us working and updating these assets. The animation video was created with open source tool Gource.
This article is part of a series on monitoring.
In the old days we just had to manage a single mainframe and a private network with terminals and printers. Now technology has evolved into service oriented, REST-api based, lightweight, runs in the cloud and communicates with tens of other services. Developing these services takes a week and deployment is fully automated. So the Agile DevOps factory is spitting out new services every minute.
This results in a higly dynamic environment and it becomes a real challenge to manage thousands of micro services. How do you get an overview on the current state of your system. That is what the next video explains.
Nowadays the large companies and institutions have complex IT Services to support business value chains that execute millions of business transactions around the clock. All this business activity need to be monitored for problems and technical failures.
Question: How can we assure 99.7% availability and reduce cost at the same time ???
In this article I will describe how a typical company monitors their IT services and how it can achieve situational awareness.
Een mooie demonstratie over de belevingswereld van een robot. De robot word aangestuurd vanuit de ontwikkelomgeving Wolfram Mathematica en geeft daar ook zijn sensor data weer naar terug. Zo krijg je een goed beeld wat er zoal om gaat in het brein van de robot.
Mijn SoccerRobot gebaseerd op de V-REP robot simulator is weer een stap verder. Het speelveld is nu uitgerust met een echt goal, de robot is een eenvoudig model met twee wielen. De simulatie berekent een pad vanaf de huidige positie van de robot tot een punt vlak bij het doel. Hierbij word rekening gehouden dat de robot langs de verdedigers moet rijden.
In een volgende versie kijk ik naar een plan in drie fasen:
- Zoek de bal
- Ga naar de bal
- Neem de bal mee naar het doel
- Positioneer je voor een goed schot
- Schiet in het doel
Om snel en goedkoop te starten maak ik gebruik van de V-REP Robot Simulator. De V-REP software is ontwikkeld door een professioneel bedrijf maar is nu open source en gratis te gebruiken. Hier is mijn eerste versie van een robot die wellicht kan gaan voetballen. Hij is nog jeugdig en volop in training dus nog even geduld.
Waarom mijn interesse in robots:
- Ze zijn leuk en interessant
- Ik heb een opleiding eletronica gedaan en wil daar weer wat mee gaan doen
- Wellicht heeft mijn dochter interesse in robots en programmeren
- Misschien kan ik andere kinderen hiervoor interesseren (code-dojo)
The xPerience is an annual technology market at ING. This was an excellent opportunity for me to present Smart Monitoring an innovative solution for end to end monitoring.
More than 500 engineers, product owners, chapter leads and managers visited the event that was all about experiencing the capabilities and approaches that help us building the next generation digital bank.
Waar komen al die internet bankieren storingen toch vandaan en wat kan je er er aan doen?
Veel bedrijven richten hun aandacht op het snel ontwikkelen en in de markt zetten van nieuwe diensten. Deze wens naar snelle ontwikkeling word ondersteund door de Agile werkwijze waarbij scrum ontwikkelteams continue en in korte sprints van enkele weken nieuwe componenten opleveren. Door diverse oorzaken raakt het beheer van de bestaande diensten wat op de achtergrond en de kwaliteit van de geleverde diensten neemt af. In dit artikel wil ik “Smart Monitoring” introduceren, een werkwijze voor het efficiënt en effectief monitoren en verbeteren van stabiliteit en beschikbaarheid van IT services.
My current assignment at ING is coaching and training several people in three different DevOps teams to fullfill the SMT role. DevOps teams at ING develop and manage SOA services based on TIBCO ActiveMatrix / BusinessWorks. The teams work according to the Scrum method and Continuous Delivery Engineering Practices. This means short sprints with multidisciplinary teams. The primary tasks of a SMT person (Service Managment Team) are to deploy and test new services, resolve issues and handle incidents. This also includes daily operations tasks.
- Alles onder controle hebben
- Incidenten effectief afhandelen
- Rustig kunnen slapen
- Verantwoording naar de business (SLA)
- Verantwoording naar de klant (kwaliteit)
- Kwaliteitsverbetering (dienstverlening, resource gebruik)
- Terugkoppeling naar de leverancier (projecten)
The ProRail InfoPlus system provides round the clock information to travellers. The system processes thousands of messages per hour and delivers this over a TIBCO Enterprise Service Bus to external parties such as OV9292. Mr Ritter is asked to assist sub-contractor Conclusion Future Infrastructure Technologies in delivering 7*24 Gold support. Conclusion will provide remote support on the TIBCO Enterprise Service Bus. Mr Ritter is responsible for preparing the remote support team for this important task.
Now I have my smart phone a HTC Desire, I seem to have desire for more. A smartphone is not really smart. It is just a mobile computer with a bunch of small applications. Ok, its good for keeping track of you appointments and email but it will not do anything to help you on its own. It is just not context aware, so i wanted to change that.
How much do we really know about our business processes:
- Number of started / completed / aborted processes
- Success / failure rate
- Turnaround time
- Root cause of failures
During my latest customer engagement i used some basic process mining and analysis techniques. Process mining is a process management technique that allows for the analysis of business processes based on event logs. The basic idea is to extract knowledge from event logs recorded by an information system. Process mining aims at improving this by providing techniques and tools for discovering process, control, data, organizational, and social structures from event logs. Continue reading
TIBCO Enterprise Service Bus provides a complete set of ESB products, these products need to be combined into a solution architecture. For my current customer I designed and implemented the following monitoring and reporting solution that resulted in a significant reduction of business process exceptions.
- OpsView (Enterprise IT Monitoring)
- TIBCO Hawk (monitor infrastructure behavior, metrics and failures)
- TIBCO Clever (monitor functional and technical errors)
- TIBCO Spotfire (reporting)
- Pentaho Data Integration (ETL)
- Esper (Complex Event Processing)
- Confluence (Wiki based knowlegde base)
Why do we need it?
Monitoring and Reporting is not just an add-on to the business functionality that can be added long after the design of business functionality has finished. It is an essential element of the architecture and design. It provides you with all the tools and information required to run business process as expected.
Once you have setup a Social Network with an active community it is time to take a better look at what goes on inside the network. Every network has been setup to serve one or more goals like community building, knowledge sharing and social interaction. How can we determine if our Social Network is sucesfull and how to steer it in the right direction? Continue reading
Currently i am working on setting up a webshop based on open source software. The frontend is a Magento webshop, in the backend OpenERP will handle fulfilment and administration. OpenERP is a wonderful system that handles sales, procurement, warehousing and accounting. This is an exciting project.
The Saint Lucas intranet is now un-officialy live. After a period of software installation, trials and experimentation the time has come to open the intranet to the community. The first user groups have been setup and the early adopters registered themselves.
Rijkswaterstaat is the executive arm of the Dutch Ministry of Infrastructure and the Environment. On behalf of the Minister and State Secretary, Rijkswaterstaat is responsible for the design, construction, management and maintenance of the main infrastructure facilities in the Netherlands. The IT department of Rijkswaterstaat has identified the need for a Enterprise Service Bus and has requested Capgemini to deliver a solution within six weeks.
Mr Ritter was asked to take care of the design and implementation of the Enterprise Service Bus and deliver a complete solution based on Mule ESB. The project was a real challenge because of the short delivery time.
- Implement an Enterprise Service Bus based on Mule ESB, JBoss Application Server and HornetQ Messaging
- Install the software in a VMWare Redhat Linux environment
- Define guidelines and best practices
- Provide documentation (installation, maintenance, development)
- Designed and build integrations:
- Idenity management (OpenAM)
- Electronic data exchange (Digikoppeling)
The project was delivered just in time.
A parish with eight locations, 100 groups and over 500 volunteers has recognized the need for their volunteers to share their knowledge and experience.
Mr. Ritter started to investigate the potential benefits of a Social Intranet and finally implemented a solution based on the open source Elgg system.
During my vacation in Thailand I got interested in teaching and e-learning.
During one of my previous visits to Thailand I had been teaching English classes in local schools. My newest assignment requires me to teach IBM WebSphere MessageBroker skills to a small group of my customers employees. This led me to the question how can we share knowledge with other people for learning purposes. I found moodle a very interesting open source solution.
The One Canon Programme of Canon Europe has the ambition to redesign and improve its worldwide IT services and at the same time lower cost by outsourcing Functional Application Management. The new IT solutions where developed by multiple suppliers (Capgemini, Logica, Deloitte and IBM) and the new IT environment was outsourced to Capgemini India. Because of a best of breed approach several applications needed to be integrated with middleware IBM WebSphere Process Server and ETL tools.
Mr Ritter was asked to participate in this programme because of his experience with middleware and outsourcing.
Tasks and duties:
- Collect and catalog all existing and missing knowledge and documentation on the newly build middleware solutions
- Support the knowledge transfer to a Managed Resources team from India
- Define requirements for a error handling and monitoring system
- Responsible for setting up and providing production support during the transition period
- Handle incidents (HP Openview)
- Analyse and fix problems in WebSphere Process Server code (mediations, Java programming)
- Create daily reports on production errors in ETL tooling
Do you know that 80% of all data in the world is unstructured. Think about email, office documents, blogs and social media (Tweeter). Im now investigating how to use Natural language processing technology to extract interesting facts from data. The project that im working on now involves scanning requests for job positions and determining what specific skills are needed. Im using the following tools:
- Open source NLP software from GATE
- Protege for building a background knowledge base (Ontology)
- IBM LanguageWare
I will update article soon. If you want to know more take a look at the GATE project.
This webinar describes the various deployment options for Apache Camel.
Learn how to easily use and leverage the power of Apache Camel and Enterprise Integration Patterns in a range of deployment options varying from plain java applications to full blown SOA implementations.
Designed from the ground up to be an easy-to-use integration framework, great value is provided to any developer with a need to integrate by providing a breath of connectivity options and implementations of the Enterprise Integration Patterns. By not relying on a heavy specification as starting point, Apache Camel has no dependency on a runtime container and has virtually endless deployment options.
Please join us for this free and interactive webinar that explores the deployment options of Apache Camel, whether you want to deploy with a messaging backbone such as FUSE Message Broker, a full ESB such as FUSE ESB or within your own plain Java or Java EE application this webinar covers the various deployment options available.
This webinar includes:
• Apache Camel basics and deployment options
• demo with code samples
• interactive Q&A
• how to use open source software to cut costs
• how to get enterprise-class services and support
The presentation will introduce FUSE Mediation Router, a productized and fully supported distribution of Apache Camel that is designed for enterprise use. Be sure to join us for this informative and interesting session.
On December i attended the IBM Deep Dive session in Behoust near Paris. I was send there with some other collegues to learn about the new integrated IBM Process Server product offering.
Agility and alignment off business and IT are two top-priorities of our customers. Business Process Management Suites (BPMS’s) offer us the possibilities to realize wishes of our customers business organization, in short time. BPMS’s are changing and will change the way we build systems and integrate them. BPMS is one of the top-priorities of Capgemini in the Netherlands. Capgemini is the ideal partner for customers and for our technology partners to do projects with BPMS’s. This because we have thorough knowledge and experience with business processes, architecture and technology.
On the 13th and 14th of December you have the possibility to Deep Dive into the BPMS technology of IBM which consists of IBM Websphere Process Server, IBM Websphere ILOG and IBM Websphere Lombardi. This Deep Dive is the first one of a sequence that will be organized by the cluster Java Software Integration & BPMS of the division Custom Software Development.
A special Tiger Team of IBM, flown in from the US, will learn you about the strategy of IBM in this area and will show you an overview of the IBM BPMS stack. Demo’s will provide the real Deep Dive. In the margin of this Deep Dive IBM will show you H2 recognize BPMS opportunities and H2 pitch at customers.
The objective of this conference is to give participants insight in the latest developments and practical applications of Open Source software in building integrated business solutions. The program will consist of independent keynotes as well as of contributions that are based on real project- and client experiences to demonstrate how using Open Source technology can leverage extensive business value.
Where: Muziekgebouw aan ‘t IJ, Piet Heinkade 1, 1019 BR Amsterdam, The Netherlands
- Tomas Nyström, Senior Director Accenture
- Werner Knoblich, VP & General Manager Red Hat
- Ron Tolido, VP & CTO Capgemini
- Bryan Cheung, Founder & CEO Liferay ; Eric Kluijfhout, Open Universiteit
My comment: I specialy liked the analysis by Accenture on the usage of Open Source software and the speech of Eric Kluijfhout on Electronic learning based on the Liferay application. The Open Universiteit deploys Liferay to develop and market networked and community-based services in the field of life long professional development at higher vocational and academic level.
I love Thailand. The first time I was there i joined a small group tour, visiting cities and some adventurous three day walk in the jungle. Later I went all by myself, its very easy to find your way around. People are very friendly and helpfull. During one of these trips I met my wife.
The partnership between Capgemini and IBM goes a long way back. To show the strength of the combination of Capgemini’s professional services combined with IBM’s revolutionary hard- and software, the Capgemini-IBM Alliance organizes the Big Blue Night on October 14, 2010 at Capgemini Congress Center in Utrecht. This event is open for customers, Capgemini professionals and selected IBM professionals and will start at 17:00 hrs with welcome drinks. A great finger food buffet will be served at the information market starting from 19:30 hrs. The evening ends when you decide to leave.
After the keynote session the attendees have the opportunity to attend two out of a choice of seven breakout sessions. Afterwards you will have the opportunity to stroll the information market, enjoy great food and drinks and the music of The Quinted.
Attended the IBM “WebSphere Update” meeting in Amsterdam. Got some more interesting news on WebSphere Application Server and the DataPower appliances. Also did some networking and met Richard Krajicek our national tennis hero.
“During the inspiring sessions at the WebSphere Update event you will discover how you can operate more dynamically and cost-efficiently by using these advanced tools. You will learn how to make opportunities grow. How to accelerate projects. How to take charge of challenges. In short, you will find out how to grow from ‘regular’ to optimum business performance. Enzo Greco, IBM World Wide BPM expert will share insights into how the business environment is changing and how you can get started working smarter. You will discover the fundamental capabilities that will help drive agility into your organization and enable your company to outperform.”
My comment: “Learned about the Datapower appliances and had a photo shoot with Richard Krajicek who enjoyed standing next to me”
On 8th September 2010 Capgemini organizes another Cloud Computing Conference: Proven Innovation. We gladly invite you to attend this conference. During this event, companies using cloud solutions will share their experiences and you will be informed about how they created business value with cloud computing.
Cloud computing in practice: Business value and pitfalls Enterprises who started to use cloud computing solutions within the last two years will share their experience with you and will inform you about new innovations they are working on. They will share the benefits and added value that cloud solutions brought their enterprise, but also the concerns about and pitfalls of cloud computing.
Meet our CTO The Global CTO of Capgemini , Andy Mulholland, co-author of the book: ‘Enterprise Cloud Computing, a strategy guide for business and technology leaders’, will share his insights about enterprise cloud computing, ‘… for the Wow isn’t just about technology, it’s about the promise of on-demand business innovation’.
Binckbank staat voor de uitdaging om het aantal verschillende ontwikkelomgevingen bestaande uit Microsoft .Net en Java terug te brengen tot één gestandaardiseerde omgeving.
De heer Ritter is gevraagd om voor deze situatie een oplossing te vinden. Dit resulteerde in een advies voor het inrichten van de SOA Infrastructuur met TIBCO producten gebaseerd op de TIBCO ActiveMatrix software. Deze oplossing bied de mogelijkheid om binnen één beheerplatform zowel de bestaande Microsoft .Net als Java systemen te ondersteunen en geleidelijk naar één implementatie technologie te migreren.
BinckBank is an online bank for investors with a Top 5 position in Europe. They have the ambition to become the biggest online bank for investors in Europe, measured by the number of brokerage accounts, transactions and profit, and aim to be active in a minimum of five countries. BinckBank’s activities are split into two business units: ‘Retail’ and ‘Professional Services’. BinckBank has a daughter called Syntel. Syntel develops software for banks, pension funds and insurance companies. With its EuroPort+ product, Syntel has been a market leader in the field of investment applications for many years.
Binckbank faces the problem that it has several development environments in place, this results in extra costs and hinders service reuse . Mr Ritter was asked by his colleague Hans van de Koppel to advice him as a Solution Architect, investigate how to improve this situation and advise on a strategy. After several interviews we found that the client has multiple business units that have developed components in Java, Microsoft .Net, Delphi and Magic using different types of technology for communication.
A report is presented to the board with a plan to transform the current situation into a more integrated solution based on a SOA architecture supported by a single deployment tool.
I’m investigating the capabilities of Amazon EC2 Cloud (EC2 means Elastic Cloud Computing). Cloud computing in general promises us direct access to large amounts of servers for a reasonable price. I think Cloud computing offers a great opportunity for my customers because a project can set up an environment within minutes instead of weeks or even months. For Amazon EC2 you needed to get an account, that is easy to do within 5 minutes when you have a mobile phone and credit card. But amazon is not the only player in the field.
Gartner Magic Quadrant for Web Hosting and Hosted Cloud System Infrastructure Services (On Demand), 2009: