SQL server powershell war chest – advanced connection provider

in a previous post I covered a basic connection provider, which is nice, but has its limitations. biggest one being that you always have to supply credentials. If you’re like most DBAs, chances are that you usually work on several servers in different environments and entering the login info all the time can get tedious. Of course integrated authentication helps a bit, but still, for those that still use SQL logins it could be a bit of a nuisance. All of this was a problem until my friend Jorge Satorre showed me a beautiful solution that I will share with you here.
First of all, let’s start from the basics – saving your password for future reuse. This blog post may come in handly. Jorge likes Powershell Cookbook (I believe it’s this one). In any case, first step is to save a password to a file. It’s a bit of a tango twist, but not too confusing. First we create an object with the following properties – AuthenticationType, Username and SecurePassword. AuthenticationType can be integrated or regular:

$object = "" | Select-Object AuthenticationType,Username,SecurePassword

the twist happens when saving a password. We take it from a regular string to a secure string:

convertto-securestring "P@ssW0rD!" -asplaintext -force

then we encrypt the secure string for storage and export to an xml file:

$object = "" | Select-Object AuthenticationType,Username,SecurePassword
$object.AuthenticationType = "regular"
$object.Username = "superuser"
$object.Password = convertto-securestring "P@ssW0rD!" -asplaintext -force | ConvertFrom-SecureString
$object |  Export-Clixml ./ServerA.xml

we can create a file for each server and store them in a save location. As you will see in the connection script I store my server authentication files in c:\powershell\ss.
We’re almost done. Now our get_connection script just needs two params – a database name and a server.

param(
	  [string] $database = "MYdb",
	  [string] $dataSource="ServerA"
     )

all we have to do is to read the credential file and construct a connection string accordingly. If the AuthenticationType is “regular” we will also read the username/password pair.

       $Path = "c:\powershell\ss\$dataSource.XML"

        # Import credential file
        $import = Import-Clixml $Path 

if ($import.AuthenticationType -eq "Integrated")

{
$authentication = "Integrated Security=SSPI;"
}
else
{
        # Test for valid import
        if ( !$import.UserName -or !$import.Password ) {
                Throw "Input is not a valid ExportedPSCredential object, exiting."
        }

		$username = $import.Username

		        # Decrypt the password and store as a SecureString object for safekeeping
        $SecurePass = $import.Password | ConvertTo-SecureString

        # Build the new credential object
        $Credential = New-Object System.Management.Automation.PSCredential $Username, $SecurePass

	    $plainCred = $credential.GetNetworkCredential()
    	$authentication = ("uid={0};pwd={1};" -f $plainCred.Username,$plainCred.Password)
}

$connectionString = "Persist Security Info=True;Initial Catalog=$database;Data Source=$dataSource;$authentication"

that’s it. Now all of your servers are at your fingertips. Download the get_connection script.

Automatically deploying data dude projects with vsdbcmd.exe

In a previous post, I talked about automatically building a database project outside of Visual Studio. Just for the sake of a feeling of completeness, I will also add a script for deploying the said project to a database of your choice.

There are many ways to deploy a database project. See for example this msdn reference for examples of deploying without having to specify a connection string on a command line, or Justin’s blog on using TFS build templates, or jamie thomson on using msbuild scripts.  For my part, I’m driven by a desire to accomplish as much as possible with doing as little as possible. So, after I run the build script and all is well, I want to re-use the build output files without modifying them at all.

Build the output directory should contain the following files:

  • dbname.dbschema
  • dbname.deploymanifest
  • dbname.sqlcmdvars
  • dbname.sqldeployment
  • dbname.sqlsettings
  • dbname.PostDeployment.sql
  • dbname.PreDeployment.sql

they should contain all the info that we need to run vsdbcmd.exe. we’ll just need to add few params:

  • connections string
  • manifest location
  • target db name

that’s all really. We’ll need to add some basic params just so that vsdbcmd.exe knows what we want from it, such as:

  • action (deploy)
  • deploy to database
  • verbose/silent mode

As you may have guessed, I’m going to use powershell to run it:

 $connStr="Persist Security Info=True;Initial Catalog=""dbname"";Data Source=127.0.0.1;uid=""user"";pwd=""password"";"
$pr1="/Action:Deploy";
$pr2="/dd:+";
$pr3="/cs:"+$connStr
$pr4="/manifest:"""+$buildOutput+"\"+$dbname+"\"+$dbname+".deploymanifest"" "
$pr5="/p:TargetDatabase="""+$dbname+""" "
$pr6="/q:+"  

Check out this msdn page for more info on individual parameters. $buildOutput  is the path to the build output files. If I want to deploy a bunch of databased, I could turn $buildOutput  and $dbname into parameters and loop through the project directories. But  I don’t find myself deploying many databases at once. If an environment really needs to be refreshed, it’s better to do it from a CI tool, so that there is a record of action and so on.

You may have noticed that I put each param into a separate variable. This is because of a strange quirk of Powershell – vsdbcmd.exe combination. You can’t just create one long string with parameters and execute vsdbcmd with it. Vsdbcmd will think that the entire string is an /Action command. From a cmd batch file it works fine.

In any case, all’s left is to specify a path to vsdbcmd and execute it

$ex="C:\Program Files\Microsoft Visual Studio 10.0\VSTSDB\Deploy\vsdbcmd.exe"
#execute the deploy command
& $ex $pr1 $pr2 $pr3 $pr4 $pr5 $pr6;

That’s it really. As simple as it gets. If something went wrong vsdbcmd will complain about it. If you want to be a bit more proactive about it, you can follow Gert D’s   suggestion with a Powershell twist:

#check if succeeded or not
if ($?) {Write-Host "database ",$dbname," successfully deployed"}
	else {Write-Host "deployment for ",$dbname," failed"}

Download a sample powershell script.

building your database solutions outside of Visual Studio

Visual Studio database solutions provide DBAs with a lot of functionality not the least of which is ability to spot errors before scripts are deployed. One thing to be aware, though that there are three levels of error catching -

  • as soon as a file is saved, VS checks syntax and references
  • at build time references are re-examined and more thorough checking is conducted
  • during a deployment execution related errors are surfaced

the first two steps roughly correspond to what happens when a create object script is executed in the Management Studio.

So logical thing to do, of course, would be to re-build you DB solution as you’re making changes. if you have Continuous integration that runs the build process for you that could be one option, but normally it takes time until the build server processes your solution. Especially given that DB solutions tend to take longer than let’s say C# projects. You can of course re-build right in your VS studio, but that means a lot of thumb twiddling or lots of coffee runs.

One simple answer is to use msbuild.exe to re-build your DB “from outside”. I’ve been using a poweshell script to do this (surprise) with a lot of success. It gives me instant visual ques and provides a log file that I can use for further troubleshooting if needed. and also I can use the output for deploying the database to an actual SQL server using vsdbcmd.exe. After having read a great blog post on this subject by jamie thomson aka SSIS Junkie I felt compelled to share my build script. Here it is:

#MSBUILD.PS1
# automate build

[string]$output=""

$logpath="c:\PowerShell\log\buildLog.txt"

if (Test-Path $logpath)
{Remove-Item $logpath}

#output path
$buildOutput="C:\PowerShell\output"

$dbnames=@()

$msbuild="C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\msbuild.exe"

 $cdt=Get-Date

 #save output to a file
 $output=$output+ "script executed at " + $cdt + "`r`n"

 #also display the same interactively
 Write-Host "script executed at "  $cdt

 $output=$output +"`r`n"
 Write-Host

#load project list
 dir . -r  -include *.dbproj |where {$_.Name -notmatch "server"} |select-object fullname,name |
 % {

 $path=$_.fullname
 $dbnames +=$_.Name.replace(".dbproj","")
 $buildOutput=$buildOutput+"\"+$_.Name.replace(".dbproj","")

#for each dbproj file display its name
 $output=$output+ "********** building " + $_.fullname + " ********************" +"`r`n"
 write-host "********** building "  $_.fullname  " ********************"
 $output=$output+"`r`n"
 Write-Host

#execute the build and save output
 [string]$output1= &$msbuild $path /nologo /verbosity:n /p:OutputPath=$buildOutput

 $output=$output+$output1 +"`r`n"
 $output=$output+"`r`n"
 $output=$output+ "********** Finished building " + $_.Name + " ********************" +"`r`n"

 #check if the build succeeded
 if ($output1.contains("Build succeeded"))
 {
 Write-Host "Build Succeeded"

#display any warnings
$output1 | select-string -pattern "\d+?\sWarning\(s\).+$" |%{$_.Matches[0].Value}
 }

 #maybe it failed?
 if ($output1.contains("Build FAILED"))
 {
 Write-Host "Build Failed" -foregroundcolor red

 #display build warnings
 $output1 | select-string -pattern "\d+?\sWarning\(s\).+$" |%{$_.Matches[0].Value}
 }
 Write-Host
 Write-Host "********** Finished building "  $_.Name  " ********************"
 $output=$output+"`r`n"
 $output1=""
 $buildOutput="C:\PowerShell\output"
}

 # save output
 Out-File -filepath $logpath -InputObject $output

to download MSBUILD.ps1 as a file.

SQL server powershell war chest – connection provider

It is clear by now that MS SQL server and powershell are going to be good friends for a long time to come. So a prudent DBA should be building a war chest of script modules that can be reused in different cases. I will go through a series of short posts that will make every DBAs life easier and more productive.

let’s start with basics – connection provider. If you’re going to interface with a SQL server, you need to connect to it. Here is a very simple script that will do the job:

#get_conection.ps1

param(
      [string] $datasource="myserveraddress",
	  [string] $database = "MyDb",
	  [string] $authType = "Integrated",
	  [string] $username ="",
	  [string] $password=""
     )

## Prepare the authentication information.
if ($authType -eq "Integrated")
{
$authentication = "Integrated Security=SSPI;"
}
else
{ $authentication = ("uid={0};pwd={1};" -f $username,$password)	}

# Create the actual connection string
$connectionString = "Persist Security Info=True;Initial Catalog=$database;Data Source=$dataSource;$authentication"

## Connect to the data source and open it
$connection = new-object System.Data.SqlClient.SqlConnection $connectionString
$connection

As you can see, there is practically nothing to it. you can easily switch between SQL [regular] and Windows [integrated] modes of authentication. We’ll be using this script a lot in the future.

Download Get_connection.ps1

Working with suppress warnings in Visual Studio Database Solutions

MS Database Projects have come a long way from their humble beginnings especially with GDR and now full integration into VS2010, but plenty of issues remain. One of the more persistent problems is tracking references. Sometimes we just have to repress those warnings (code 4151). (http://msdn.microsoft.com/en-us/library/aa980442(v=vs.80).aspx) :

setting Suppress Warnings 4151
 

 

that’s all fine and good and gets us out of a bind short term, but if you really are going to use your DB project as intended, you should keep the number of objects that need this to a minimum. One problem is that once you set it, there is no visual way to identify a “suppressed” object. One way out of it is to parse the dbproj file. With Powershell, of course, it doesn’t get any simpler. Here is a script that I use. I’ve added comments for clarity:

#H4151.ps1

$root="."
#check if the output file exists, if so delete it
IF (Test-Path c:\temp\output.csv)
{Remove-Item c:\temp\output.csv}

# Create a file header
$output="DBProject,SQLObjects `r`n"

#get the files
$files=(dir $root -r | where {$_.extension -match "dbproj"}|Get-Item)
foreach ($file in $files)
{
#get the file naame
$ProjectName=$file.Name
$ProjectName

#get the second ItempGroup block
$fileString=Get-Content $file
$text_list=[regex]::Matches($fileString,"<ItemGroup>(.*?)</ItemGroup>") | Select-String "SuppressWarnings" | %{$_.line}

#let's add a root element to the list
$text_list="<root>"+$text_list+"</root>"

$xd=1$text_list

#get all the object Names
$xd.SelectNodes("/root/ItemGroup/Build[SuppressWarnings=4151]") | %{
$objName=($_.Include).split("\")[-1]
write-host $ProjectName,$objName;
$output=$output + $ProjectName + ","+ $objName+ "`r`n"
}

}

Out-File -filepath c:\temp\output.csv -InputObject $output -Append

Download a sample powershell script.