Business Intelligence Integration Joel Da Costa, Takudzwa Mabande, Richard Migwalla Antoine Bagula, Joseph Balikuddembe Project Description Business Intelligence (BI) is the practice of using computer software to aid data analysis and decision making in businesses. It represents a set of processes, tools and technologies which improve productivity, sales and service of an enterprise, and so profitability in general. BI works primarily by collecting, organizing and analyzing corporate data and then creating useful knowledge out this analysis (reporting). BI as a whole incorporates a wide spectrum of software functions including ad-hoc querying, on-line analytical processing (OLAP), dashboards, scorecards, search, visualization and more. BI differentiates itself through its interdepartmental focus and general overview which is geared towards total business performance. The implementation of BI gives knowledge and understanding to departmental groups which previously may not have had access to or understanding of the data. Increased analytics and ad hoc reporting allow organisations to better understand trends within their business and apply a variety of different measures and attributes to understanding these trends. Once the BI system has been implemented, a company will typically find it has more ideas for new initiatives, more efficient and precise data collection processes, more effective marketing techniques and a better understanding of its customers‟ needs and characteristics, and a better understanding of the state of the market. This improved business agility and efficiency through BI results in a long term performance gain which can result in significant profit increases. The BI system itself is typically segmented into several key areas. The first is Business Modelling in order to create the framework of the system and how the information flows need to be established. Data warehouses are used as a centralized repository for all the data gathered, and maintained through the 'Extraction, Transformation and Loading' (ETL) processes. OLAP is a technique by which the data sourced from the data warehouse is visualized and summarized to provide a perspective view across multiple dimensions in order to quickly answer multi-dimensional queries. Essentially, OLAP tells a business what has happened, and Data mining explains why it happened, and what is likely to happen in the future based on past patterns. Problem Statement The project is going to focus on the underlying technologies which enable Business Intelligence (BI) and their application to two key scenarios. Previously, various technologies have been developed and implemented from a “one size fits all” approach, but this approach is likely to result in less effective and accurate analysis. Different areas and analyses require more adaptability rather than such a singular approach. Our aim then is to evaluate which technologies would be the most effective for the particular cases. The technologies being evaluated are Bayesian Belief Networks, Neural Networks and Artificial Immune Systems which will be expanded on later in the proposal. The project will be done in cooperation with Sanlam, who will provide the necessary data to be analysed. The first case is to analyse customer data in order to create profiles of them so that they may be targeted with the correct marketing techniques. By doing so, it would allow an increase in sales as well as a decrease in the cost of marketing. Using the data provided, the 3 different technologies will be applied to try and gain the most accurate customer profiles. The second case is „Predictive Sales Forecasting‟. Using historical and current data, the same 3 technologies will be applied to try and create an accurate forecast of the future trends. Forecasting allows better business decisions to be made and mitigation to be taken in order to improve the likely outcome. While the cases operate individually, they are all being implemented with the same aim. What we want to ascertain from the project results is the variance of each approach's results when measured against the same data and also bench-marked on the known sales figures. This will help define the strengths and weaknesses of the particular technologies in developing BI functions. Procedures and Methods This project is primarily designed as a research venture, with the main objective being the synthesis of usable research results, as per the Problem statement. The scope does however extend beyond that, as shall be illustrated through the following breakdown of procedures and objectives. Implementation will occur in the form of a java application. It will make use of 3 different intelligent systems to analyse historical data provided by Sanlam, to predict the required output. Rationale for chosen approach Before considering the actual approach that will be used in addressing the Problem statement, it is first necessary to mention the reasoning behind the choice of algorithms. As the research conducted indicated, industry tends to favour the use of these 3 algorithms, particularly exhibiting a distinct liking to Bayesian Belief networks. Furthermore, as per the initial meeting with the Sanlam representative, these 3 algorithms are of particular interest to Sanlam. More detail as to industry‟s use of these algorithms can be seen in the related works section. Thus, the next step is to elaborate on the chosen approach, i.e. the choice to address the problem in the form of an application. The following points summarize the motivation behind this: Application development allows further extensibility: By choosing to develop this project in the form of an application, there is more room to generalize and adapt the application, making it useful in other spheres of Business. Extensibility also allows room for improvement. Thus, developing this application allows room for continuation, evolution and progress. Providing Sanlam with a concrete showing of the results obtained, as well as how they were obtained is also reason for developing this application. Recreating the results of this research experiment will also be made easier given the platform of an application. Development process Because of the collaborative nature of this project, it is key that the primary stakeholders i.e. Sanlam, submit a clear description of their requirements and expectations. For this reason, the project will involve a Sanlam delegate, as well as the project team. Meetings will be held with the delegate in order to generate a specific set of user requirements from which the solution can be derived. Once the requirements have been finalised, the next phase will then be implemented. This will consist of developing the application which will model abstractions of the selected intelligent systems. The application will have various forms of clientele information, (provided by Sanlam) as input. This information will include elements such as Incomes, Premiums as well as purchasing history, to name a few. Based on this input, the application will then use the embedded intelligent systems to generate output, offering the user the choice as to which algorithm is applied in the simulation. This functionality will thus allow for comparison of results. The output will be displayed in a format that is relevant to business users, and a graphical user interface will be implemented as part of the application. By hiding a significant portion of the underlying technicalities, and displaying only what is relevant to shareholders and other business analysts the interface will thus achieve its functionality (more detail on this is provided later). Ethical, Professional and Legal Issues Ethical Issues We will be using Sanlam sales and customer data which is to remain confidential. It may not be redistributed to any external parties and no personal information may be extracted for use outside the project. For demonstration purposes, the software may not display personal information that may lead to the identification of particular individuals. This information may be used in the generation of results/forecasts but it will be abstracted with the use of IDs for names if necessary. Legal Issues All sales and customer data from Sanlam must be kept private within the realms of the project. Any copies of the database must be deleted once testing is completed and may not be archived outside of Sanlam. No copies of the database may be created for use outside the project for any purpose. Related Work Customer Profiling Sebastiani et al. used Bayesian Networks to profile customers in order to predict profits. They used two networks: the first to describe the probability of response from customers, and the second to model price factors. The results were reasonable, and by understanding the characteristics of customers, the models thus help to potentially increase profits . Similar work has been done by Elalfi et al. who combined Bayesian networks with genetic algorithms. An algorithm was used to extract accurate and comprehensible rules from a database using trained artificial neural networks, which in turn were trained by genetic algorithms to find the optimal values for the model. These rules were then used to define customer profiles in order to make for more profitable e-business . Customer life cycles Baesens et al. introduces a measure of a customer‟s future spending evolution that might improve relationship marketing decision making. The method suggested predicts whether a customer will increase or decrease spending from their initial purchase information. It had a 75% classification accuracy in predicting the customer lifecycle using purchase volume and purchase category . Repeat Purchase Modeling Baesens et al. focuses on the need for companies such as mail-order companies to identify which customers are most likely to purchase before they send out costly catalogues. This involves profiling customers according to several parameters and calculating the probability of repurchase. A Bayesian Neural Network was used and had a correct classification result of 71% given the data set used . Modelling Customer Attitudes Ishigaki et al. use Bayesian networks to model customer attitudes based on questionnaire data. The model can then be used to gauge customers‟ feelings towards a product, and how they should be marketed to. The model was fairly successful with a 73.5% success rate on testing . Sales Forecasting Recently, Chang et al. developed on the idea of sales forecasting by including clustering in the model. The K-mean technique is used to cluster the data, which is then used with a fuzzy neural network, which once trained, can generate sales forecasts. The model proved very effective in providing accurate forecasts, and was more accurate than a series of other models it was tested against . Anticipated Outcomes We will create a package that will read in data from the Sanlam database, use different machine learning techniques to profile customers and compare the accuracy of the different techniques using actual data. System The software will be composed of: An interface to the database that will read in relevant data. The core of the program that will contain three different Intelligent System techniques that a user can utilize. The front-end interface that will give the user results of the classification comparing actual data to inferred information. The major component will be the implementation of the different techniques. However, it will still be important to have good interfaces with the database and the user. The user interface will need to display interpretable information on the performance of each technique, which will entail aggregating the results in a way that a user will quickly and easily understand. It will need to allow changes in parameters to allow optimisation for particular data sets. Expected Impact We expect to identify the best machine learning technique to use for customer profiling and sales forecasting for Sanlam in particular. From our initial investigation it seems that Bayesian networks are very good classifiers (useful in customer profiling) and neural networks are very good forecasters. The performance of each technique however is highly dependent on the task, data and results required. This may mean that the performance results in Sanlam‟s case will not necessarily match the results for other organisations/companies. Key Success Factors The results of the simulations will need to be compared to existing data of what the simulations are trying to predict. The comparisons will be used to rank each technique according to accuracy of its results. All simulations will be expected to complete within an acceptable time frame (performance and scalability are out of scope for this project but each implementation will need to run within a determined acceptable time, thus making performance negligible in determining the best technique to use). Project Plan Risk Management The risks that follow are to be evaluated based on the following risk Matrix Probability Low Medium Disastrous C B Impact Serious D C High A B Marginal E D C Trivial F E D The following table gives a breakdown of the predicted risks associated with this project, paying special attention to their impact and probability. It also highlights 2 courses of action: Avoidance that is an on-going process as well as mitigation should the risk materialize. Risk Matrix Avoidance Evaluation Loss of a project team member. (This would occur if one or more members abandoned the Honours Programme for any number of reasons) Have sufficiently D. Serious/ Pressure to stay on the independent deliverable Low project as failure to do so modules for each team Probability means not graduating. member. 2. Delay in Delivery of test data. (Dependent on Sanlam for DataExternal factor) C. Pressure Sanlam to Disastrous/ provide data as soon as Low possible. Probability Create random test data or use alternative available data. 3. Scope creep (Plan too many tasks, Cannot complete tasks in time) E. Marginal/ Project planned in detail Low with supervisor and Probability department approval. Start with fundamental features first and leave other things to the end. 1. 4. Data loss due to hardware failure, (External Factor) C. Serious/ Medium Probability Frequent backups of all progress on different machines or storage devices. C. Serious/ Medium Probability Review and reassess Constant reference to the deadlines; readjusting project timeline and clear where necessary- as communication between cost-effectively as project members possible. 5. Missing project deadlines 6. Mitigation Misunderstanding User requirements. D. Serious/ (Resultant of Low miscommunication/ Probability ambiguity in user-team interaction) Constant communication with Sanlam to maintain correct direction. Also, providing Sanlam with project plan and design in order to detect flaws. Roll back to last backup. Iterations through development so that inconsistencies can be detected early. Timeline & Gantt Chart Resources Required The resources required to complete the project are fairly standard, with the software and equipment in the Honours Lab sufficing for development. Apart from this though, Joseph Balikuddembe is necessary as a representative of Sanlam and as co-supervisor for the project. Furthermore, the data regarding customers and sales that Sanlam will provide is crucial to the project development. Necessary Resources: PC‟s Sanlam Database Access Java Development Platform Deliverables The following Table illustrates a detailed list of the deliverables necessary for the completion of this project: Deliverable: Final Project Proposal Project Proposal Presentation Project Web Presence Project Poster Project Web Page Project Report Project Application Description: Final copy of Proposal for evaluation. Presentation to supervisor and class. Online availability of proposal and project timeline. Poster representation of Project. Open Availability of Project Webpage. A report on the results of the research. The actual project. Further detail as to dates can be seen as per the Milestones. Milestones Milestones Literature Synthesis Project Proposal Project Proposal Presentations Finalized Project Proposal Project Web Presence Prototype Background/Theory Chapter Design Chapter Database interface setup Customer Profiling BI Techniques Sales Forecasting BI Techniques Visualization/GUI First Implementation Final Prototype Chapters on Implementation and Testing Outline of complete report Final Complete Draft of Report Poster Web Page Reflection Paper Project Demonstrations Final Project Presentations Dates Mon 3-May Wed 12-May Mon 17-May Mon 31-May Tue 1-Jun Fri 4-Jun Fri 22-Jun Mon 6-Jul Tue 20-Jun Tue 24-Aug Tue 14-Sep Fri 17-Sep Mon 20-Sep Wed 29-Sep Mon 4-Oct Mon 11-Oct Mon 25-Oct Thu 04-Nov Mon 8-Nov Fri 12-Nov Wed 03-Nov Thu 18-Nov Work Allocation Joel Da Costa will implement the Bayesian Belief Networks algorithm for the two cases. Additionally, he will handle the necessary implementation of connecting to, or drawing data from the Sanlam database. Takudzwa Mabande will implement the Neural Networks algorithm for the two cases. Additionally, he will handle the usage of data for the Sales Forecasting case, as well as the output visualization for both cases. Richard Migwalla will implement the Artificial Immune Systems algorithm for the two cases. Additionally, he will handle the usage of data for the Customer Profiling case, as well as the general GUI implementation. References  Sebastiani P., Ramoni M., Crea A. Profiling your Customers using Bayesian Networks. SIGKDD Explorations 1(2). 91 – 97.  Elfalfi A., Haque R., Elalami M. Extracting rules from trained neural network using GA for managing E-business. Applied Soft Computing 4. 65-77  Baesens, B., Verstraeten, G., Van Den Poel, D., Egmont-Petersen, M., Van Kenhove, P. And Vanthienen, J. 2004. Bayesian network classifiers for identifying the slope of the customer lifecycle of long-life customers. European Journal of Operational Research 156, 508-523.  Baesens, B., Viaene, S., Van Den Poel, D., Vanthienen, J. And Dedene, G. 2002. Bayesian neural network learning for repeat purchase modelling in direct marketing. European Journal of Operational Research 138, 191-211.  Ishigaki T., Motomura Y., Dohi M., Kouchi M., Mochimaru M. Knowledge Extraction by Probabilistic Cognitive Structure Modeling Using a Bayesian Network for Use by a Retail Service. MEDES October 2009. 141-149  Chang P, Lio C, Fan C. Data clustering and fuzzy neural network for sales forecasting: A case study in printed circuit board industry. Knowledge Based Systems 22. 344- 355.
© Copyright 2020