Category: technology

technology

Wiggle It Spindle IT

chicpencil

The hard worker is always behind the scene. In the case of Online Transaction Processing (OLTP) and Decision Support Systems (DSS), what’s not seen in the naked eye, the hard worker is the disk subsystem which always remains unnoticed.   ~ Chic Pencil

A customer relationship management system will always contain and maintain our records once we become a member or a customer of the business entity. Customer records is always the basis of customer analytics as well as customer loyalty campaigns.

When you want your records to be retrieved, what you know is to input your personal details when it is done via online portal or a customer service officer will retrieved it for you.

Say for example you input your details as the following via the online portal:

  • Firstname      :     John
  • Lastname       :     Smith
  • Date of birth  :     12/08/1980
  • National ID   :     589001122

In the backend processing, the CRM system will get your records via passing of your inputs to the CRM database. As such, the CRM system will pass the query using the following SQL statement (the language of most databases):

> begin tran

> select * from customer where fname = ‘John’ and lname = ‘Smith’ and DOB = ’12/08/1980′ and NID = 589001122

> end tran

> go

Once the database receives your request, it will be placed in the queue. And when it is your request’s turn for the database to process, well the database will do the following:

  1. The database will parse the SQL statement received. The parsing process involves checking of syntaxes and if there are violations, the database will return the errors to the CRM system.
  2. Once the parsing completes and there’s no violations found, the database will refer to its query processing engine to build the algorithm that will retrieve the data. We call this stage, the building of the query plans.
  3. Based on the query plans generated, the database will choose which plan will fit to retrieve the data. The query plan is one of they key workers of a database system. It is essential that the database administrator (DBA) performs the necessary maintenance in order to keep the database system at optimal health.
  4. The chosen query plan will be performed by the database engine. This time, the database engine will perform the coordination with the last worker in the hierarchy — the disk subsystem.
  5. When the disk subsystem receives the database request, it will place it in a queue. And the queue management and control is handled by spindle. The spindle will tell the disk to get the requested data basing from RPM (speed) of the physical disk.
  6. Once the spindle completes the requested operation, it will pass it back to the database with the data in it.
  7. The database will then relay the requested data to the CRM system.
  8. The CRM system will display the data requested. In this case it will validate based on the input if the customer does exists in the customers table.

The above example is only looking for data in one table. When you want to know the purchases you made or any transactions you done in the past, the database system will look for your query in different tables and indexes. The more requests you do translates to more operations to the last worker in the hierarchy and that is the disk subsystem.

Though the role of the disk subsystem is sometimes forgotten. It does play a major role in any IT needs from personal to enterprise requirements.

The disk spindle holds the key role in letting the IT system administrators know the input – output (I/O) speed — MB/sec of the subsystem it manage. It is the benchmark of the system administrators if there is a need to increase or upgrade the current disk subsystem when new business requirements arise.

Most often than not, the importance of the having the best disk subsystem is limitedly appreciated by the hardware vendors. Most software vendors does not put emphasis that by having a best disk subsystem their system will perform par better than their expected speed.

The disk subsystem is the database’s best friend. When the database is fast in getting user queries, the disk subsystem should be able to response fast as well otherwise, choking on the disk system will occur. Disk choking occurs when the disk spindle is not able to meet up with the transactions that is passed on to it. Remember that the transactions not only comes from database requests but also from the operating system side as well.

As such, for OLTP requests, a higher disk RPM and a higher disk spindle is an excellent fit. This requirement is universal regardless of database brand. These days, 10K or 15K RPM SAS disk is readily available in the market.

For DSS or OLAP requirements, you will need to consider the database architecture and technology. Since user queries especially ad-hoc queries requires huge amount of I/O operations as data retrieval is being done in bulk method, parallelism inclusive of disk spindle would be an excellent fit. Well the database technology for this requirement will be discussed in a separate article.

For Unix system, you can view the disk spindle (operations) via these commands

  • vmstat
  • sar

It is important to monitor the health of the disk. Once this component fails, a RAID configuration should be handy otherwise precious data will be completely loss as a backup system may not be able to do an up-to the minute backup.

So, if you’re not taking care of your disks better start checking it now. It is the worker who wiggles and spindles in order to retrieve your important data and files and everything in your system.

Exposure 1/60 sec. F/4.5. Focal 70 mm. ISO 400. Nikon D200.

Title: Simply happy. A symbol of a happy customer.

A happy customer is a loyal customer. Keeping customers happy not only entails business processes but also the IT system as well. As such, everything in the equation has to be balanced. And the last worker in the hierarchy shouldn’t be forgotten.

Til next time. Au revoir.


analytics, business, cep, technology

The Value of Complex Event Processing (CEP) & The Story of Alice in the not so ever Wonderland

chicpencil

Once upon a time there was a lady (let’s call her Alice) who was very practical that she has her fixed deposit – FD savings in tact in the bank. The bank is very happy with her savings as it would give the bank the assurance and increase in their wealth management for greener investments.

The bank is equipped with a high-end infrastructure in analytics and sophisticated business intelligence (BI) system giving their business analysts and frontline business professionals flexibility in their quantitative analysis, trending, and forecasting.

The BI infrastructure of the bank is still using the traditional approach. Incremental and delta data changes will be extracted by the ETL, that is, Extract, Transform & Load the data to their target analytics engine. Or the approach can be  ELT, that is, Extract, Load, & Transform the data to their target analytics engine. Once all the transformations are complete, the data that are once OLTP in nature now becomes OLAP and is ready for analysis and ready for usage of the bank’s downstream systems.

The service level agreement (SLA) of the bank’s IT department to the business is 12 hours. The extraction of data up to the completion of all the transformation jobs needs to complete at 6AM everyday. Previously, the SLA was 15 hours since the decision systems environment is free by 3PM. But due to ever changing demands of business analytics, the SLA is now reduced to 12 hours and the IT department is having tough time coping up because their analytics database engine is starting to suffer from performance degradation.

The bank’s business analysts does collect their data everyday but their method of analysis based on the business process is within 1 week worth of data. There wasn’t a near real time business process to analyze the behavior of account holders.

Alice’s FD account has been very good for the last 2 years. She belongs to the demographics of single, female, high income professional, very good standing on accounts, and within the age of 25 to 30.

However changes happened in Alice’s life. In less than 1 week, Alice performed the following transactions:

  1. Withdrawal from FD accounts
  2. Withdrawal from Savings accounts
  3. Withdrawal from Current accounts
  4. Applied for a personal loan

After the above withdrawals (Events 1 to 3), her demographics has actually changed to single, female, high income professional, poor standing on accounts (risk), and within the age of 25 to 30.

When Alice applied for the loan (Event 4), she was surprised that her loan was granted. But of course, she was happy.

Why did the bank approved Alice’s loan?

Simply because when the loan officer saw Alice’s account holder profile, she still belongs to the very good accounts. Though there were withdrawals from Alice’s accounts, the loan approver thinks that Alice may have some investments in the bank that is not visible to the loan officer due to the bank’s security policies.

The loan officer doesn’t know that the demographics profile analysis wasn’t updated yet.

Clearly based on the above scenario, Alice is under the risk of defaulting her payment to her loan and the bank is also under the risk for non payments.

If all the four complex events that Alice did were captured in real time and handled by various business processes or rules, the bank would have the chance to acknowledged the risk of lending the loan to Alice to avoid financial losses.

Alice story is a normal customer behavior.

However, if there are one million bank customer who performed the complex events that Alice did without the bank realizing it, it is not surprising why banks suffered financial losses and can probably attributed to the recent financial downturn that happened globally. The Great Depression of the Financial markets.

After the business realizes their gaps, financial institutions have demanded for real time analysis of events.

Previously, the OLAP is sitting in the datawarehouse and is only accessed and analyzed once a week or everyday for operational reporting, the business demands have changed that they need the events to be processed and analyzed as soon as it happened and entered in the company’s analytics system.

Analytics has evolved. As always.

Complex events processing or greatly known as CEP systems is the analysis of different levels of events that correlates to the business rules or processes.

As soon as the events have entered into the CEP stream, it is analyzed immediately based on the rules thus triggering the different actions as provided by the business logics. It provides advancement and flexibility to the company’s frontline force – business analyst and business support team.

What’s the value of having a CEP system?

(1) It provides an enhanced risk management system. The earlier alert of Alice’s complex event 4 would trigger the bank that her demographics has changed thus the loan should be disapproved.

(2) Fraud detection. When I was in Oman, I purchased the jewelry set and used my Singapore credit card in payment. In less than 5 minutes while I was still in the jewelry shop, the bank called me to verify the validity of the transaction. As CEP systems analyzed the near real time events even in seconds, considering that I was overseas and with few delays due to the network speed, etc, the complex event that I triggered changed my behavior pattern thus it is suspect for fraud detection as I have never been in middle east let alone use my credit in middle east. The good thing about it is that if it was a true fraud, the jewelry shop can hold the person who is misusing the credit card since the bank will reject the transaction. Thanks to CEP systems.

What happens to traditional OLAP models?

Well, the usage of ROLAP, MOLAP, HOLAP, etc is still going to be used by the business for their trending, slicing and other analysis for better business positioning as required by the C-levels of the company. However, the CEP systems is catered for frontline force and business support team for real time business activity monitoring.

Who offers CEP systems in the market?

There are large enterprise vendors offering CEP solutions. Below is the list to name a few and this list doesn’t follow any ranking:

So, next time when a bank calls you to verify your transaction, don’t get annoyed especially if you’re not in the mood. You have to thank your bank that it is investing in this type of technology to better service you as a valued customer. (Wink).

Good morning America. Photo taken in New York, USA. 2005.

Til next time. Au revoir.

analytics, business, competitive advantage, technology

You’re Tagged And Analyzed

chicpencil

You walked in a supermarket looking for a pack of tissue as you are probably suffering from colds or for variety of reasons you need to get a tissue from that supermarket as it is the nearest store from you according to your location finder services or digital maps and GPS.

However, you spent thirty minutes walking around the supermarket searching for the tissues section and you walked away empty handed as you went impatient and just decided to search and try your luck from other shops.

You ended up frustrated and the supermarket may have loose their opportunity for you to be their loyal customer.

If the supermarket would have just placed some retails of facial tissues within the cashier counter you would have noticed it and bought it and you will praise the marketing of the supermarket as they have done good job and satisfied your need at that moment.

Analyzing what happened would lead us to realize that there was the gap between the customer and the business. There is the probability that the supermarket didn’t have a built in analytics system in their process.

Analytics – the process of solving problems based on existing set of data has been used by technology to solve business problems and create competitive advantage.

Majority of the business players invest in analytics software in order to analyze their customers, create trends, and most of all discover new demographics.

Caesar Salad with grilled chicken at Mc Donald’s in Amsterdam, The Netherlands. Now this encounter is impressive.

It is all about consumer behavior and patterns.

And technology has been evolving consistently to support the business demands.

Previously the analysis has been based on structured data – age, location, behavior, race, etc. However, as the business competition is getting tougher, technology has evolved and now supports unstructured data for analysis. The term ‘unstructured data’ is referring to blogs and videos to name a few.

Now the method of consumer analysis can start

  1. From the moment that you walk in the supermarket looking for your items.
  2. From the items that you need you are choosing from different brands. While choosing you may have been analyzed by the brand names that implements the video analytics.
  3. When you pick up the item it is a different record of event and
  4. when you decided to buy the item it is tagged as different event.

When it is the first time that you choose your credit card to purchase online, your bank suddenly called you in a matter of minutes to confirm if you really made the transaction. That is analytics in the move and the technology supports it to analyze it in milliseconds or seconds to provide fraud detection.

As all of these are happening it is a great feeling that our technology can keep up and business is there to provide us with better service and satisfaction. The downside of which is the issue on privacy as consumers.

Who are you? This is my nephew giving me his inquisitive look during my birthday celebration in Germany, September 2010.

Though privacy issues created commotion due to internet cookies detected to collect surfing data and other personal data, it is still undeniable that our personal data maybe roaming somewhere being analyzed as which demographics we belong and what regression model would suit and fit us so the business can create advantage and maintain competition.

Privacy issue is always undeniable in the digital world. And our behavior or pattern are the key to business and as well as government in providing security and welfare.

And so, we are tagged and being analyzed.

And if we do not want to be located for whatever reason… the best way to do it is to shut off the mobile phone. (wink).

This is me while gearing up and competing with in the market space of analytics technology.

Til next time. Au revoir.