SQL Bits @ Brighton

foto

As promised, here are my slides and the demo code.

I had a great time in Brighton and I’m looking forward to the next one, can’t wait to find out where it will take place !

Don’t just load the demo code and run it. I used a table with real data at the end which is obviously not in the demo script. But you should be able to read through it and use my code.

 

The script and the powerpoint can be dowloaded here :

Powerpivot for DBA's

Virtual File Stats Demo

If you have any questions, just let me know.
If you were in my session, please fill out the eval !

T-SQL Tuesday #005 – “Book Index” styled SSRS Report

Note : This post is part of T-SQL Tuesday #005, this months subject is “Reporting” and it’s hosted by Aaron Nelson (@sqlvariant).

I needed an overview report with the names of all the applications we are responsible for. It was intended for the application support teams who could find their applications name in the report and click on it to be forwarded to another report with data on their application.

I found that a report styled like the index of a book worked well for this situation.

This demo is based on the vendor list of the Adventureworks demo database, I’m creating a view that get’s the list of “applications” from the Purchasing.Vendor table.
I also needed a table with the letters of the alphabet.

Create table Alfabet(Letter char(1))
Insert into Alfabet(Letter) values('A'), ('B'), ('C'), ('D'), ('E'), ('F'), ('G'), ('H'), ('I'), ('J'), ('K'), ('L'), ('M'), ('N'), ('O'), ('P'), ('Q'), ('R'), ('S'), ('T'), ('U'), ('V'), ('W'), ('X'), ('Y'), ('Z')

Create view Application
as
Select
	VendorID as ApplicationID,
	Name
From Purchasing.Vendor

To get the correct resultset for my report I built a stored procedure.
After creating the temp table I fill it with a list of applications, using a CTE to make 4 equal(ish) parts.

Create table #AppList(ApplicationID int, Application varchar(50), isLetter int, NTileNr int, RowNumber int)

;with ApplicationList
as
(
	Select
		NULL as ApplicationID,
		Letter as Application,
		1 as isLetter
	From Alfabet
	union all
	Select
		a.ApplicationID,
		a.Name,
		0
	From Application a
)
Insert into #AppList(ApplicationID, Application, isLetter, NTileNr)
Select
	ApplicationID,
	Application,
	isLetter,
	NTILE(4) over(Order by Application)
From ApplicationList

After that I add a rownumber per part to be able to show them side by side in the result set later on.

;with ApplicationList
as
(
	Select
		Application,
	ROW_NUMBER() over(partition by NTileNr order by Application) as RowNr
	From #AppList
)
Update #AppList
SET RowNumber = RowNr
From ApplicationList
Where #AppList.Application = ApplicationList.Application

And finally we create a resultset by returning the application names in 4 columns accompanied by the ID which we need to link to the detail report and an isLetter field that will help us determine if this particular value came from the Alphabet table. This is useful because I’m going to apply a different style to those letters in the report.

Select
	App1.ApplicationID App1_ApplicationID,
	App1.Application App1_Application,
	App1.isLetter App1_isLetter,
	App2.ApplicationID App2_ApplicationID,
	App2.Application App2_Application,
	App2.isLetter App2_isLetter,
	App3.ApplicationID App3_ApplicationID,
	App3.Application App3_Application,
	App3.isLetter App3_isLetter,
	App4.ApplicationID App4_ApplicationID,
	App4.Application App4_Application,
	App4.isLetter App4_isLetter
From(Select Distinct RowNumber From #AppList) AppList
Left Join (Select RowNumber, ApplicationID, Application, isLetter From #AppList Where NTileNr = 1) App1 on App1.RowNumber = AppList.RowNumber
Left Join (Select RowNumber, ApplicationID, Application, isLetter From #AppList Where NTileNr = 2) App2 on App2.RowNumber = AppList.RowNumber
Left Join (Select RowNumber, ApplicationID, Application, isLetter From #AppList Where NTileNr = 3) App3 on App3.RowNumber = AppList.RowNumber
Left Join (Select RowNumber, ApplicationID, Application, isLetter From #AppList Where NTileNr = 4) App4 on App4.RowNumber = AppList.RowNumber
Order by AppList.RowNumber

Drop table #AppList

Next step is to create a report, this is what it looks like after simply starting with the wizard :

Wizard

After that there’s a couple of things that need to be done to make it look better :

  • Remove the column header row
  • Make the “Index” title larger en center the text
  • Make the columns wider
  • Create an extra column on the left and leave it empty
  • Switch the border style of the grid to none

Now it should look similar to this :

AfterCleanup

The final step is to make the letters more visible. We can set the text box properties for all the columns. We can use expressions to change the horizontal alignment and the style and size of the font of the column only where it needs to have a different view because it’s a letter and not an application name.

Expression

Enter the following expressions, make sure the AppX_isLetter column has the right number of the column for every column in the report.

  • Alignment – Size –> =IIf(Fields!App1_isLetter.Value=1, “Center”, “Default”)
  • Font – Style –> =IIF(Fields!App1_isLetter.Value=1, “Bold”, “Normal”)
  • Font – Size  –>=IIF(Fields!App1_isLetter.Value=1, “14PT”, “10PT”)

And we’re done (almost) :

EndResult

Action 

The final thing we need to do is enable the Application field as a hyperlink :

The thing I don’t like here is that I would love to be able to have an expression ability for the “Enable as a hyperlink” feature as well. Now the application is clickable even if there is no detail available and the letters can also be clicked on…

Note : You can name your Alphabet table “Alphabet” instead of the Dutch “Alfabet” if you like, just make sure the query knows about this change as well :-)

The scripts and the demo report can be dowloaded here :

BookStyleIndexReport.sql

Index.rdl

SQL Zaterdag is Dutch for SQL Saturday

It’s somewhat similar but actually our first SQL Zaterdag is inspired mostly by SQLBits, if only ours could one day become half as successful as the event from our friends in the UK.

DSCN0091

For now, I’m happy. This Saturday we are expecting around 200 people which is the maximum the venue will hold. I’ve done a survey and it looks like most people will show up (no-shows are the biggest problem of free events, we do have to pay the caterer…).

And according to the survey the track preference is around 65% – 35% for respectively the BI Track and the Engine Track.

Looking good so far, lot’s of things to do still.

Wish us luck…..

Find SQL Servers on your network with Powershell

Update : 23-01-2011 – Stephen Spence noticed a bug in the script, versions of SQL Server 2000 below SP3 would not be identified.
I’ve updated the script below, it now identifies down to the gold build of SQL Server 2000 but will give “unknown” for some of those older versions as a type.

This powershell script will scan the registry of servers remotely.

$infile = ".\" + $args[0] + ".csv"
$outfile = ".\" + $args[0] + "_out.csv"

$ErrorActionPreference = "SilentlyContinue"
"Server, Instance, Version, Edition" | Out-File $outfile
foreach ($Server in import-csv $infile)
{
  $server
  $version = $null
  $reg = $null
  $reg = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey('LocalMachine', $Server.Name)

  $regkey = $null
  $regkey = $reg.OpenSubKey("SOFTWARE\\Microsoft\\Microsoft SQL Server\\Instance Names\\SQL")

  IF ($regkey){
  foreach ($regInstance in $regkey.GetValueNames())
  {
    $regInstanceData = $regKey.GetValue($regInstance)
    $versionKey = $reg.OpenSubKey("SOFTWARE\\Microsoft\\Microsoft SQL Server\\$regInstanceData\\Setup")
    $version = $versionKey.GetValue('PatchLevel')
    $edition = $versionKey.GetValue('Edition')
    $VersionInfo = $Server.Name + ',' + $regInstance + ',' + $version + ',' + $edition
    $versionInfo | Out-File $outfile -Append
  }}

  IF ($version -eq $null)
  {
    $regkey = $null
    $regkey = $reg.OpenSubkey("SOFTWARE\\Microsoft\\Microsoft SQL Server")
    IF ($regkey){
    foreach ($regInstance in $regkey.GetValue('InstalledInstances'))
    {
      IF ($regInstance -eq "MSSQLServer")
      {
	$versionKey = $reg.OpenSubkey("SOFTWARE\\Microsoft\\Microsoft SQL Server\\MSSQLSERVER\Setup")
	IF ($versionKey.GetValue('PatchLevel') -eq $null)
	{
          $versionKey = $reg.OpenSubKey("SOFTWARE\\Microsoft\\MSSQLServer\\Setup")
          IF ($versionKey.GetValue('PatchLevel') -eq $null)
          {
            $versionkey = $reg.OpenSubkey("SOFTWARE\\Microsoft\MSSQLServer\\MSSQLServer\\CurrentVersion")
          }
 	}
      }
      ELSE
      {
        $versionKey = $reg.OpenSubkey("SOFTWARE\\Microsoft\\Microsoft SQL Server\\$regInstance\\Setup")
      }
      IF ($versionkey)
      {
        $version = $versionKey.GetValue('PatchLevel')
        $edition = $versionKey.GetValue('Edition')
        IF ($version -eq $null)
        {
          $version = $versionKey.GetValue('CSDVersion')
          IF ($version -eq $null)
          {
	    $version = $versionKey.GetValue('CurrentVersion')
	  }
        }
	IF ($Edition -eq $null){$Edition = "Unknown"}
        IF ($version)
        {
          $VersionInfo = $Server.Name + ',' + $regInstance + ',' + $version + ',' + $edition
          $versionInfo | Out-File $outfile -Append
        }
      }
    }}
  }
}

What you need is a list of servers.
It should have a csv extension and the first column should say Name.

serverlist_csv

Then simply run the script giving the filename without the csv extension as a parameter :

ScanSQLServers

The result will be written to a file with the same name as the input file but with _out.csv :

serverlist_out

Do you have a lot of servers, and you want to know how many you need to patch ? :

serverlist_groupversion

This relies on the Remote Registry service running on the server you are scanning and you also need to have sufficient rights (local admin). I’m using it myself to scan the boxes that I manage since I’m a local admin on each one. But to get an overall scan done of the entire network I’ve asked a domain admin to run it for me and send me the resulting out_csv.

(I’ve created a list of servers for him to scan using this method)

The script is tested starting with SQL Server 2000 and al the way up to 2008 R2
(2000 stores the info in the registry in a different location from later editions of SQL Server, which is why the script is as long as it is.
However, it did find a frightening amount of SQL Server 2000 servers at my customers site :-)  )

Parsing dsquery output with Powershell

Getting a list of all servers from your active directory is simple, and looks something like this :

dsquery

I wanted to get the list of servers (dc, sql1 & sql2).
In Powershell this can be done by splitting the line 3 times.

This looks like this :

@(dsquery computer -limit 0
      | %{$_.split("`"")[1]}
      | %{$_.split(",")[0]}
      | %{$_.split("=")[1]})

@ is used to call the dsquery process.
% is short for for-each

  • So, split on “  (the backtick is the escape character) and taking the second value from the resulting array is the first step and removes the quotes.
  • After that, split on , will break the line up in key/value pairs where only the first value from the array will be used (CN=…)
  • Finally split on the = and take the second value in the resulting array. This leaves you with a list of servers….

Want to filter out stuff ?

?{$_ -like '*OU=Servers*'}

use this after the first pipe symbol an pipe the results into the for-each statements.
The ? means where-object or where.
This example shows you only the OU’s named “Servers”.

The PASS Summit is not a holiday

My colleagues keep asking me if I had a nice holiday.
I’m getting tired of explaining that it’s serious business and therefore decided to post some pictures of a bunch of us engaged in the best SQL Server training event around….

IMG_7131 IMG_7132 IMG_7133 IMG_7134 IMG_7135 IMG_7136 IMG_7138 IMG_7139 IMG_7141

Reading Packed Decimals with SSIS

Update 16-10-2010 : I’ve updated the code because we found a bug regarding the precision when dealing with small amounts.
(This code has been used in a production system for over a year now, but do use any code you find on the web at your own risk !)

Have you ever got files to import from a Mainframe ? Chances are that the file is in some ebcdic encoding, and could contain something called Packed Decimals.
(also known as COMP-3 in Cobol)

You can find an explanation of what a packed decimal is here.
Basically it’s a compression method for decimals.

Let’s take for example the string “esdfsm”, this takes up 6 characters in a text file.
But when you convert this to hex you get “65 73 64 66 73 6D”.
Now we have 11 digits, the last character in the hex representation tells you if it’s positive or negative. An F means positive and the D stands for negative.
There is no place to store the decimal separator, but if you know what the original type was, you can just insert it in the right place.

If this were a decimal(11,2) you would get –657.364.667,36

Note: The ebcdic encoding often just shows weird unreadable characters, which need to be encoded from ebcdic to unicode first before you can read them or insert them in SQL Server. (SQL Server does have some ebcdic encoding support, but not for ebcdic(500) which is a common ebcdic International encoding.)
You can’t encode the whole file though because the packed decimals will lose their meaning, these need to be calculated from their binary form.

Reading such a file with SSIS can be done using a Script Component.
Just follow these steps :

Note: This example is based on a fixed length file, you could have a CSV file and would need to change the code to your needs….
Also, the packed decimal will be transformed to a right aligned string.
(I needed that for my situation, so as with the csv example, you need to change the code if you need a slightly different representation.)

1. Add a script component to a data flow and choose “Source” when asked for the Script Component Type

image

2. Add the appropriate output name and column names and types. I’m exporting everything as a Unicode string in this example. The packed decimal is the SaleAmount field which is 13 characters wide. (6 bytes becomes 12 hex, containing 11 digits and a sign. +1 for the decimal separator).

image

3. Go to the script tab, press Edit Script… Add the following code. (Replacing the existing code)

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.IO;
using System.Text;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{

    public override void PreExecute()
    {
        base.PreExecute();
        /*
          Add your code here for preprocessing or remove if not needed
        */
    }

    public override void PostExecute()
    {
        base.PostExecute();
        /*
          Add your code here for postprocessing or remove if not needed
          You can set read/write variables here, for example:
          Variables.MyIntVar = 100
        */
    }

    public override void CreateNewOutputRows()
    {
        Encoding ebcdic = Encoding.GetEncoding(500);
        Encoding unicode = Encoding.Unicode;

        using (BinaryReader b = new BinaryReader(File.Open("c:\\ebcdicfile.dat", FileMode.Open)))
        {
            byte[] line;
            int pos = 0;
            int length = (int)b.BaseStream.Length;

            while (pos < length)
            {
                line = b.ReadBytes(45); // Make sure to specify the exact length of 1 line in the file here !
                pos += 45;

                MyDataBuffer.AddRow();

                MyDataBuffer.CustNo = unicode.GetString(Encoding.Convert(ebcdic, unicode, line, 0, 4));
                MyDataBuffer.CustName = unicode.GetString(Encoding.Convert(ebcdic, unicode, line, 4, 35));
                MyDataBuffer.SaleAmount = UnPack(line, 39, 6, 2);
            }
        }
    }

    public string UnPack(byte[] Data, int Start, int Length, int Precision)
    {
        string unpacked = "";
        string signed = " ";
        int pos = 0;

        while (pos < Length)
        {
            unpacked += Data[Start + pos].ToString("x2");
            pos += 1;
        }

        if (unpacked.EndsWith("d"))
            signed = "-";

        unpacked = unpacked.Remove(unpacked.Length - 1);
        unpacked = unpacked.TrimStart('0');

        if (unpacked == "")
            unpacked = "0";

        if (Precision > 0)
            if (unpacked.Length > Precision)
                unpacked = unpacked.Insert(unpacked.Length - Precision, ".");
            else
                unpacked = "0." + unpacked.PadLeft(Precision,'0');

        unpacked = signed + unpacked;

        if (Precision > 0)
            unpacked = unpacked.PadRight((unpacked.Length + Precision) - (unpacked.Length - (unpacked.IndexOf(".") + 1)), '0');

        if (Precision == 0)
            unpacked = unpacked.PadLeft(Length * 2);
        else
            unpacked = unpacked.PadLeft(1 + Length * 2);

        return unpacked;
    }

}

 

4. Change the appropriate code in the CreateNewOutputRows() function to reflect your file specification.

Change : File.Open(“c:\\ebcdifile.dat” to reflect the correct file location
Change : b.ReadBytes(45); and pos += 45; to reflect the length of a line in your file.
Change : MyDataBuffer to reflect the output name you have chosen (SSIS adds “Buffer” to the end)
Change: CustNo, CustName & SaleAmount to your own fieldnames.
Change: (ebcdic, unicode, line, 0, 4) etc. where the last values represent the starting position in the file and the length
Change: UnPack(line, 39, 6,2) etc. where the 3 values represent the starting position in the file, the length in the file and the desired number of decimals.

5. Save, close the script window and press OK.
Now all that is left to do is read the data into the database with an OLE DB Destination.

image

SQLCAT team at Dutch PASS Chapter – Swimsuit Edition

Our Dutch chapter was honoured by a visit from the SQLCAT team. Lubor Kollar and Thomas Kejser from the SQLCAT team even brought Sameet Agarwal and Amit Shukla along who are from the Core Engine team for SQL Server.

IMG_6927

IMG_6928

 

 

 

 

 

 

 

The evening was hosted by Henk van der Valk who works for Unisys. Our chapter meets every month at various locations, we depend on the generosity of company’s who sponsor our user group meetings. Unisys has become a true “friend of PASS” in the past years having sponsored a number of meetings already, largely due to fellow member Henk who runs the Performance Test Centre of Unisys in the Netherlands.

The setup of this evening was pretty simple but sure did the trick. Lubor introduced Thomas, Amit, Sameet, himself and explained what the SQLCAT team is and does. After that Sameet explained what his Core Engine team is and does. The rest was up to us, we were free to ask questions after that. Following a silence of about 20 seconds was a 2 hour Q&A session. The questions were quite literally about almost everything. Complaints, feature requests or scalability / maintenance tips and tricks. You could ask pretty much anything and you would either get an answer or notice that Lubor was making lot’s of notes.

The cool part is that you can do this too. The SQLCAT team members can be found at many conferences like SQLBits, the European PASS Conference (more on the 2010 edition soon) and of course the PASS Summit. And let’s not forget to mention that they share a wealth of knowledge through white papers etc. via their website www.sqlcat.com !

Lubor showed proof at the end of the evening that you can literaly run into CAT team members anywhere by showing a picture of Kenneth Willhelmson, himself and yours truly from the SQL Server Open World conference in Lalandia, Denmark.

DSCN0407

Or maybe showing this picture on the screen in front of 50 chapter members was a trick to get me to be quiet. Like Sameet calling me a trouble maker was not a strong enough hint. Well, they did say that you could say anything, even complain about something or request features :-)

It was all in good fun of course and I would like to thank Lubor and Thomas from the CAT team, Sameet and Amit from the Core Engine Team, Henk van der Valk from Unisys and of course everyone who showed up from our PASS Chapter for this excellent Q&A session !

Pizza & Beer during 24 hours of PASS

IMG_6531

We actually only watched about 8 hours of PASS.

The plan was to do around 10 hours but it took 1 hour to figure out that while a HDMI delivers a great picture, Windows Live Meeting only wants to play sound through the headphone connection. And we lost an hour because we used the wrong link….

Which actually made our setup only look more geeky, because I had to use a speaker set from a pc.

I encourage everyone to invite some geek friends over next time PASS does this. The discussions and sharing of ideas was excellent ! The presentations / presenters kept on throwing ideas into the room and the informal setting made everyone bounce ideas into the group.

Besides Henk, who is just showing off, remote connecting to his 96 core box :-)

IMG_6529

I’m kidding of course !  At some point Thomas Grohser’s (=tg=) backup and restore white paper was mentioned. A print-out is laying around somewhere between the pizza boxes. And we debated on various backup and restore methodologies for a significant portion of the evening.

Thanks PASS, please do it again !

96 cores at SQLBits IV

I’ve submitted my session for SQLBits IV.

This time I’m bringing a guest. Henk van der Valk runs the Unisys test centre and we all wish we had his test box at home. Probably until we get the first electricity bill :-)

HenkThomasKeynote

The picture is from the European PASS Conference in Dusseldorf 2009, Henk van der Valk presenting together with Thomas Kejser from the SQLCAT team. Look at the task manager on the screen….

It’s a 96 core box with 1/2 Terabyte of memory (!)
During the demo in this picture it was attached to a DSI RAMSAN which can do around 600.000 IOPS….

This will be my 4th SQLBits and I’m looking forward to it again, it’s a great conference. This year they have even expanded the concept and besides  a free Community Day on the Saturday, there is also a Training Day on Thursday and a not-free-but-also-not-expensive conference day on the Friday.

Check it out at www.sqlbits.com