What the heck is Business Intelligence?

The term Business Intelligence (BI) refers to technologies, applications and practices for the collection, integration, analysis, and presentation of business information. The purpose of Business Intelligence is to support better business decision making. Essentially, Business Intelligence systems are data-driven Decision Support Systems (DSS). Business Intelligence is sometimes used interchangeably with briefing books, report and query tools and executive information systems. Continue reading

Getting control of a codebase

So recently I started working on a new codebase. I will be honest when I first saw it, it was a mess. Here are a few of the things I did to try and regain control.

I was given access to the source code on Visual Studio Team Services. However this consisted of a single commit 3 months ago. When I looked at what was running on the production server it was clear that changes were being made live with no regard for source control.

The first thing I did was commit everything that was running live into source control.

Next I created a SQL Server Data Tools (SSDT) project to keep track of all the database objects. Previously there was a folder with some stored procedures in it, but these did not match with what was currently running.

SQL ServerI now had in source control the current state of the website and the database, so I knew I could get things back to this state if I made some bad changes.

Lets start by looking at the website code I had. There was no solution file, the only way to look at the website was to setup my local IIS to run what was in the website folder. I could then use Visual Studio to “open” my local IIS website and attach to process to debug it.

Next I Looked at Default.aspx to see how the website worked. The majority of the website code was stored in the database stored procedures. After the tag there was a <% %> which contained a Response.Write(RunSP.RunStoredProcedure(Parameter1, Parameter2, …) command, which executed a stored procedure and the results of the stored procedure was the html code including any javascript that the webpage needed to display. I will be honest I have never seen any code like it. My guess is that the developer was secretly a DBA and wanted to make any web page changes by just changing how the stored procedures work.

This meant that the website is not going to do anything without a backup of the database running, and meant my SSDT project was going to be vital. However the database was in a bad state, it consisted of a fair few broken objects and SSDT would not build.

Using find I went through each of the broken database objects to find where in the code they were being used. Luckily most were referenced in commented out code, so I just removed all the broken database objects. The database could now be built. However there was a dependency on the users table of another database. (This was the developers solution to sharing logins between websites) As I was using SSDT I added a database dependency, problem solved for now.

Next I tried publishing my database. SQL CMD encountered a parsing error.  The reason for this was my SPs contained javascript eg $(document), SQL CMD uses $(DatabaseName) as variables for different database so it was getting itself confused.

My solution was to use Find and Replace to replace all the $ with ‘ + CHAR(36) + ‘

So I now have a SSDT project that builds and publishes but still no website project.

To get the website running from Visual Studio I started off creating a .Net 4 website project and added Entity Framework 5 and MVC 3 via nuget. I then copied all the website code into the new project, and with a bit of work I got it to build. Most of the work was relating to namespaces and referencing the correct one and moving the EF model from AppCode to a custom named folder. A bit of trial and error later I had a version of the website that could be run from Visual Studio.

I have not deployed my new version of the website as it needs further testing. No automated testing or even a smoke test checklist currently exist.

Visual Studio Team ServicesAs my source code is hosted on Visual Studio Team Services (VSTS), I can get VSTS to build each commit and check I haven’t broken the build. This is not that helpful at the moment, hopefully one day I will have automated tests that can be run here as well.

Wow, I feel like I have done loads with this code so far but there is loads more still to do. I need to understand more about the business processes behind the code with a hope to understand why some architectural decisions have been made. I want to refactor the code as much as is possible, I would like to remove much of the html/javascript from the stored procedures as I can’t see that there is any advantage to running a website like this. Please correct my if I am wrong.



LINQ is an acronym for Language Integrated Query, which describes where it’s used and what it does. The Language Integrated part means that LINQ is part of programming language syntax. In particular, both C# and VB are languages that ship with .NET and have LINQ capabilities.

How do I use LINQ in my C# code?

To use LINQ the first thing you need to do is add the LINQ using statement.

using System.Linq;

In your code you need a datasource, for this example I am going to use a simple array, but it can be anything eg SQL, XML etc

int[] data = new int[10] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };

Next you need a LINQ query. (Note I know the Q in LINQ means query, so I have just written query query, if you are one of those people who hates seeing PIN number you might not like this blog post.) A LINQ query is very similar to a T-SQL query, so if like me you are good with databases this should make sense to you.

In T-SQL you can have:

FROM data
WHERE num = 1

In LINQ this becomes:

var query =
from num in data
where num == 1
select num;

Finally you need to do  something with the query you have written. I am just going to print the results of my query to console.

foreach (var num in query)

What other SQL like syntax can I use?

In T-SQL you can control ordering using ORDER BY, LINQ has a similar syntax orderby

orderby num descending

In T-SQL you can use GROUP BY, to do something similar with LINQ

group num by num.Type into type
select type

This now requires a change to your foreach loop so you can list what you are grouping by and what items are in that group

foreach (var type in query)
foreach (var num in type)


So you thought joining tables was a SQL Server only thing. Think again you can do this in LINQ

var joinquery =
from cust in customers
join prod in products on prod.CustomerId equals cust.Id
select new { ProductName = prod.Name, CustomerName = cust.CompanyName };


There are loads more LINQ functionality that you can use. While writing this blog I found https://code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b which has loads of examples of different queries that you can write with LINQ.

This has inspired me to use LINQ more in my code and learn more about the different queries that could be written.

Development Annoyance

How-you-can-stop-annoyance-or-frustrations-quicklyI spend a lot of my time creating new features that simplify my companies business processes.

A good example of this is an invoicing system I created. Instead of users working off different spreadsheets and copying and pasting data between various different programs the user can click a couple of buttons and everything is done and they can move on to the next task.

I am currently expanding this system so more of the companies work can be invoiced quickly and simply.

I like working on this kind of problem.

I get to discuss with the different departments involved, finding out how they work and what could make their lives better.

I then get to look at what structures already exist in the database and get to rip out anything that makes no sense (Today I had a Customer table which has a primary key called ClientId – its now a Client table with PK ClientId) and add new structures to store new information.

I then can build a user interface so the users can interact with the data.

But the most important part of the process comes next. This is when I show the users what I have built and how it will make their lives better, they can then feed back to me questions and comments. How do I do x?, what about y?, have you thought about z?

After this very valuable feedback I can go back and tweak what I have done making it better and better until the users think it is suitable for what they need.

Now why did I call this post development annoyance?

I find it intensely annoying that after spending weeks of my time working on creating something to be used. Users either make do with my solution and never tell me the one thing that really annoys them or worse they go back to the time consuming process and never feedback the one or two changes I need to make to complete the project.

Now I am not perfect, like many IT people my people skills are somewhat lacking but apart from asking people what can you do to get the feedback you need to complete the project?

I do not have the knowledge that other departments have so I consider development a team effort that requires everyone to contribute otherwise it will not finish.

The second outcome where users return to their old ways is the most annoying. It could well be months or even years before I find out that what I created is not being used any more. By that time my memory of what I did and more importantly why I did what I did has been lost and forgotten.

So users please, please tell me what works and what doesn’t and what things I can do to improve things. I want my solutions to constantly improve and get better and I can only do that with your help.

Backing up SQL databases to Azure

I recently read a blog post by Pinal Dave about how you can backup straight to Azure Storage. The procedure he described is only available for SQL Server 2014 or later.

I won’t go into detail of this method as Pinal describes it better than I can, but the basic of it requires setting up credentials and then running a backup command that includes the URL of the storage container on Azure.

Unfortunately I am running SQL Server 2005 so this process will not work for me but it did start me thinking of what ways there might be for me to use.

The next thing I tried was Microsoft SQL Server Backup to Microsoft Azure Tool. Unfortunately I did not get this tool to work correctly on my setup. However it sounds like a flexible tool that allows compression and encryption of your backup files. This tool redirects your backup files to your Azure Storage so even if I had got it to work correctly it would not have been an ideal solution as I want local copies of my backup files as well.

After this I started to look at powershell again. Following on from my recent success with powershell I know how to connect to my Azure account so all I needed to script was copying a file from my server to Azure.

Get-ChildItem *.bak -File -Recurse | Set-AzureStorageBlobContent -Container $DestContainer -Force

This command gets all the backup files in a directory (the recurse switch looks in sub directories as well) and then pipes them to the Set-AzureStorageBlobContent command. This command uploads them to the storage container defined in $DestContainer. I have added the Force switch so that it will replace any files on Azure which have the same name.

I have only been using this script for the last few days but so far it has been working well. Now if I completely loose all data from the office I can restore from any other location using the data saved on Azure. A great improvement to my disaster recovery policy.

Database Deployment

Unmanaged Database Deployment

For the past few months I have been deploying changes to my companies database every couple of weeks or so. Over a weekend when the database was not being used I would make a backup of the database, load up visual studio, deploy the database changes and copy a couple of front-end files.

My weekends and evenings have recently become a bit more precious to me since the arrival of my son James. At suitable times to run the database upgrade I am either, asleep, about to fall asleep or be spending time with James.

This entire deployment can be set to run at a time of my choosing using SQL Server Jobs, (not sure why I never thought of doing this before). I still need to check everything is working after it has run. The job can send me an email if it encounters any problems so I can still ensure the company has the minimum of problems but has more frequent changes.

Unmanaged Database Deployment

OK so what do I need to do to set this up.Database Deployment

  1. Test all your code changes are working correctly on a backup of the database and everything is committed to source control.
  2. Backup everything that is going to be changed so it can be rolled back in case of a problem. I don’t rely on the daily backup jobs for this, I do my own. Maybe I am a bit paranoid, or maybe I am just being cautious.

    BACKUP DATABASE DBName TO DISK=’E:\SQL Backups\Filename.bak’

    I would include in the backup file name the date and time of the deployment for easy reference later on. For the front-end files these are backed up daily and only ever change during a deployment so I am going to rely on the day to day backups, plus they are in source control so in a worse case scenario I can look there.

  3. From Visual Studio create a deployment script using the publish option and save as a file on your database server.
  4. Create a SQL Server Job and give it a descriptive name.
  5. In steps create a step called backup and enter the T-SQL code above. Set this to stop on failure and continue on success.
  6. Create a second step called upgrade of type operating system with the following code.

    sqlcmd -S (local) -i “E:\SQL Backups\deploy_test.sql”

    The -S parameter is the name of your SQL Server instance and the -i parameter is the path to the sql script you generated from Visual Studio above. Set this job to quit reporting success or failure.

  7. In schedule create a date time you want the job to run, make sure this is a one time job as you don’t want it to try and run again possibly causing problems.
  8. In notifications I have set to email on completion, you can set it to email on failure only. I would prefer to know the outcome of the job regardless.
  9. Using SQL Server jobs to copy files over the network is possible but is not trivial to setup so I am going to use windows task scheduler for this.
  10. Create a job in task scheduler to copy all the ADPs or ADEs that have been changed. If the SQL job fails you will need to copy the old versions of these back but that is simple enough to do.

This process can be expanded to check code out of source control and do even more automated deployments but for now this is good enough for my purposes.

How complex are my stored procedures?

Recently me and my development team have been looking at performance and how to improve it.

complexityOne area of improvement we have identified is with our stored procedures, which we plan to rewrite. But how do we identify which are the easiest to rewrite and which are the hardest?

My first thought was maybe to count the number of times each keyword is used and rank them somehow. eg each JOIN gets 5, OUTER APPLY gets 20, each term in WHERE gets 1, and then combine that with the length of the query and how many parameters.

However doing a bit of googling I came across the following blog and sql script. This script analyses the stored procedures in terms of number of lines of code, number of parameters and number of dependencies.

The complexity is divided into SIMPLE, MEDIUM and COMPLEX.

WHEN NumberOfLines * NumberOfDependencies * NumberOfParameters < 5000 THEN ‘Simple’
WHEN NumberOfLines * NumberOfDependencies * NumberOfParameters < 10000 THEN ‘Medium’
ELSE ‘Complex’


This gives a fairly good estimate of which stored procedures are the most complex and which would probably take the longest time to rewrite.

My Love/Hate Relationship with Microsoft Access

For historical reasons I use Microsoft Access as the front-end for all my line of business databases.

Access allows you to create an ADP file that can connect to your SQL Server database. However Office has dropped support for this, so the last version of office that I can use is Office 2010.

Before I start hating on Access there are some good things about the platform. Users with very little knowledge can create a database fairly quickly. Development time is also fairly low as it doesn’t take long to create a new form, and add a few buttons which open different reports.

For the last 5 years or so I have built a range of different forms in Access, (and even if I do say so myself), I have got quite good at creating forms that users can be more productive with. Access has the concept of a subform which is a form embedded in a form, each form can be connected to a different query and can be connected to each other so it is relatively easy to create something quite powerful.

Access report are an easy way to create professional looking reports that can be sent out to clients, or used by management and they also have the concept of subreports, (same as subforms but embedded in a report), this means you don’t have to write one query to pull back every piece of data that shows up in a report.

VBA or Visual Basic for Access allows you easily do things programmatically, like run something if something is selected or a button is clicked. This is not the easiest programming language to work with, however once you have used it for a while you get used to its quirks. My favourite function is IIF, the syntax is fairly simple IIF(expr, truepart, falsepart), if expr evaluates to true the truepart is executed (executed can be as simple as displaying certain text in certain conditions), if it is false the falsepart is executed, and you can also chain them together which I have done to produce something very complex looking.

Now for the negative side of things. Access creates a binary ADP file, binary files can not be version controlled in git or subversion. This is a big problem as I want to keep track of every change I make. It also means that branching or working with others on big projects is also difficult.

One work around I have found to this problem is AccessSVN. This program takes an ADP and creates text files of every form and report. It also can create an ADP from these files as well, however I have had varying success with this feature so I prefer to just create the text files if possible. What I do after I have made a change to the ADP is to run AccessSVN, create text files and commit these to source control. You can then see what controls have been changed or added. It also helps with bugs as I have found that sometimes while testing access will apply a filter to a form, which will break things if deployed, which is very annoying.

Access can only do what access was designed to do. There are some advanced features that we would like to add to our databases. Things like sending emails or uploading reports to a website. This is not Access fault as it was never designed to be able to do such things but I am often asked about what is possible.

End of support for Office 2010 is approaching and I have plans to stop using Access but until then I need to keep supporting our existing front ends.