Delivery in day(s): 4
ICT705 Data and System Integration Proof Reading Service
The aim of this report is to make Mashup Software, which will get the location of the clinics located in the various regions of the country. These Clinics are owned by the same organization. This is achieved in three phases:
1.Use of petl module to extract and load data from csv files to table and to convert XML file into the table which is later combined. Also, regex is used to clean the missing data from the CSV file.
2.In next step a RESTful WS is designed which is based on the Python Bottle Framework, it returns data in JSON format
3.In the end, a Mashup is developed which gives the GUI to the user, where a user could enter postcode of the known clinic state an get to know the location of the clinic.
To achieve this, IDE PyCharm is used and the code is developed using Python, HTML, CSS, and JS.
2.Data Merging and Cleaning
The data given by the company has missing values, which in turn will lead to the inconsistent results. Also, the data given is in different formats which are merged later into one.To solve this issue
Files that needed to be analysed for this project:
1.Clinics.csv:This is a Comma Separated value file. In it, fictional locations of the clinics are given which are owned by the company in the different Suburbs. Figure 1 shows the data in the clinics.csv file.
Figure 1: Data in clinics.csv file
2.Location.xml: another file is an XML file in which contains Longitude and Latitude of Clinics in XML file. As it is independent of the platform so, no need to embed it with any other format. The below figure shows the XML data in the browser.
Figure 2: Data in location.xml file
Implemented method to convert XML to CSV
Figure 3: petl framework that converts the XML to CSV
As said earlier, the clinic data that is given in the form of CSV file (Comma-Delimited Values) have missing values. The Email Values are missing which have to clean before the Merging process. To make those Emails understandable Regular Expression is used, in Python Regex module is used which in turn complete the data. The figure below shows the missing data (displayed by Red arrows).
Figure 4: Missing Data
Figure 5: Data after Cleaning
Code Implementation for Cleaning process
Figure 6: Code to clean the csv file
In this project of system design, we have two kinds of Data. To use them as one they need to be processed as together. In this process, those two files are merged together as one named under “clinic_locations.csv”.In this process, both files are converted into Tables using the petl module in the python and then merged together into one file.
Figure 7: Data after Merging
Code Implementation for Merging process
Figure 8: Code for merging process
Whole code for Cleaning and Merging
Figure 9: Code Snippet
3.Implementation of RESTful Web Service
RESTful Web Service is implementedon web service architecture implemented by using a Virtual structure in a Hierarchical architecture. In this project, it is implemented by using BottleFramework. Bottle is a lightweight web (micro) framework used by python because it is simple to implement and the processing is fast. Bottle is used here because it doesn’t have any dependencies over any other Python library.
How to access RESTful Web Service
In this project, the URL used is
http://localhost:8080/getlocation?postcode=abcd (abcd is the postcode that the user will enter in the web browser to run this service)
Figure 10: Output when you run (http://localhost:8080/getlocation?postcode=2000)
To start the service there is a file called clinic_locator.py open that file in PyCharm and run it. This bottle implemented framework which will be running on default web browser. The data will be taken from the previously created csv file “clinic_locations.csv”. This web service will return the output in the format of JSON, which will be fetched later by the next process (discussed in next step). To run this process the system must have previously installed Bottle module in it.
Output in the PyCharm
Figure 11: clinic_locator.py in PyCharm
Implemented Code for this RESTful Web Service
Figure 12: Code to run RESTful Web Service (clinic_locator.py)
Output of this phase
Figure 13: RESTful Web Service Output
The Figure 12 shows the output that we get after running it in PyCharm. The output is in JSON format because it will send the data over network in a structured way. Also, JSON is the most used format to communicate between the servers sometimes web applications.
The in above figure the postcode that is passed is 2000 which is postcode of Sydney.
The output has several values
1.Clinic Name: Sydney Centre Clinic
Mashups is a hybrid application that is developed from two or more sources like web application or web pages. Mashups use those applications to create a single application or in a single graphical interface. Like in this project
1.A RESTful Web Service is made which takes input from the output of the python code written of the on system processing of the data
2.The Output of the RESTful Web Service is fed as an input to the Mashup application as JSON data and further processing is done.
1.The Created HTML web page will take input from user in the form of postcode
2.The HTML page will fetch the data from the RESTful web service created in the previous step in the JSON form and will display the location on the Google map.
For showing the Google map the Google API is used. From this API I could show the location of the clinic whose postcode is entered in the search bar.
Sources of this Mashup
There are two sources of this Mashup:
1.The clinic_locations.csv file which I have created in the first step after Data Cleaning and Data Merging processes on the Locations.xml and Clinics.csv files.
2.The RESTful Web Service That is performed in the previous step in order to start the Web Service.
Implemented code for this process
There are three programming techniques used:
Figure 14: HTML code to implement the clinic_map.html
Figure 17: Index.css
Figure 18: Style file in the CSS
Run Clinic_map.html in the browser
Figure 19: Home Screen
Figure 20: Entry Field
Figure 21: Mandatory Field
Figure 22: Server Not running Error
Figure 23: Output showing location on the map
The submit query button will directly redirect to the mail client and the subject line will already be filled with the “Clinic Enquiry”.
5.Demo Running Instructions
The project is implemented in three Phases:
1.Data Cleaning and Merging
2.Making RESTful Web Service using Python Bottle
3.Creating Mashup using above two phases
3.Bottle must be installed
For Data Cleaning and Data Merging
First, open PyCharm IDE in it
1.Open file named data_merger.py and hit configure and run
Figure 24: Click on Run
Figure 25: Click on Configurations
2.Find the specified path of the Python Interpreter
Figure 26: Configure the Python path
3.Run the Python code
Figure 27: Output
For running RESTful Web Service
The process is same for this service also.
Just Edit the path and run the application
Figure 28: Select clinic_locatior.py from the run option
Figure 29: Running the Service
For running RESTful Web Service
1.To run this the user must have an internet connection for the Google maps to fetch.
2.Now click on clinic_map.html file which will open it in your default web browser
Figure 30: Click on clinic_map.html file
1.Loading Page will be displayed
Figure 31: Open in the default browser
1.You will see the figure shown below
Figure 32: Homepage of the Mashup service
1.Enter the postcode of the clinic to find the location of the clinic
In the Nutshell, this report focuses on creating a Mashup application using which the patients can find the location of the desired clinic using the known postcode of that clinic. The postcode will search in the csv and match up with the location if the location is displayed it will be shown if not present the default location will be shown.
To achieve this, several things are done. At first, the data that is given is cleaned for missing values and then merged with the different file types. The output is then stored in a file which is used by both clinic_locator.py and clinic_map.html as an input file. This is an important step as it will change the entire result.
In the second step, a web server is designed using Bottle framework. It takes clinic_locations.csv as input and returns the JSON data which is later used by clinic_map.html to search and display the locations.
In the last step, a Mashup application is developed using above two processes. Using the files created above as input this Mashup application displays the location of the clinic by getting the postcode from the user. Also, an Enquiry button is given which will help the user to send a mail with an already filled subject line "Clinic Enquiry” in it.