Automatically deploying data dude projects with vsdbcmd.exe

In a previous post, I talked about automatically building a database project outside of Visual Studio. Just for the sake of a feeling of completeness, I will also add a script for deploying the said project to a database of your choice.

There are many ways to deploy a database project. See for example this msdn reference for examples of deploying without having to specify a connection string on a command line, or Justin’s blog on using TFS build templates, or jamie thomson on using msbuild scripts.  For my part, I’m driven by a desire to accomplish as much as possible with doing as little as possible. So, after I run the build script and all is well, I want to re-use the build output files without modifying them at all.

Build the output directory should contain the following files:

  • dbname.dbschema
  • dbname.deploymanifest
  • dbname.sqlcmdvars
  • dbname.sqldeployment
  • dbname.sqlsettings
  • dbname.PostDeployment.sql
  • dbname.PreDeployment.sql

they should contain all the info that we need to run vsdbcmd.exe. we’ll just need to add few params:

  • connections string
  • manifest location
  • target db name

that’s all really. We’ll need to add some basic params just so that vsdbcmd.exe knows what we want from it, such as:

  • action (deploy)
  • deploy to database
  • verbose/silent mode

As you may have guessed, I’m going to use powershell to run it:

 $connStr="Persist Security Info=True;Initial Catalog=""dbname"";Data Source=127.0.0.1;uid=""user"";pwd=""password"";"
$pr1="/Action:Deploy";
$pr2="/dd:+";
$pr3="/cs:"+$connStr
$pr4="/manifest:"""+$buildOutput+"\"+$dbname+"\"+$dbname+".deploymanifest"" "
$pr5="/p:TargetDatabase="""+$dbname+""" "
$pr6="/q:+"  

Check out this msdn page for more info on individual parameters. $buildOutput  is the path to the build output files. If I want to deploy a bunch of databased, I could turn $buildOutput  and $dbname into parameters and loop through the project directories. But  I don’t find myself deploying many databases at once. If an environment really needs to be refreshed, it’s better to do it from a CI tool, so that there is a record of action and so on.

You may have noticed that I put each param into a separate variable. This is because of a strange quirk of Powershell – vsdbcmd.exe combination. You can’t just create one long string with parameters and execute vsdbcmd with it. Vsdbcmd will think that the entire string is an /Action command. From a cmd batch file it works fine.

In any case, all’s left is to specify a path to vsdbcmd and execute it

$ex="C:\Program Files\Microsoft Visual Studio 10.0\VSTSDB\Deploy\vsdbcmd.exe"
#execute the deploy command
& $ex $pr1 $pr2 $pr3 $pr4 $pr5 $pr6;

That’s it really. As simple as it gets. If something went wrong vsdbcmd will complain about it. If you want to be a bit more proactive about it, you can follow Gert D’s   suggestion with a Powershell twist:

#check if succeeded or not
if ($?) {Write-Host "database ",$dbname," successfully deployed"}
	else {Write-Host "deployment for ",$dbname," failed"}

Download a sample powershell script.

building your database solutions outside of Visual Studio

Visual Studio database solutions provide DBAs with a lot of functionality not the least of which is ability to spot errors before scripts are deployed. One thing to be aware, though that there are three levels of error catching -

  • as soon as a file is saved, VS checks syntax and references
  • at build time references are re-examined and more thorough checking is conducted
  • during a deployment execution related errors are surfaced

the first two steps roughly correspond to what happens when a create object script is executed in the Management Studio.

So logical thing to do, of course, would be to re-build you DB solution as you’re making changes. if you have Continuous integration that runs the build process for you that could be one option, but normally it takes time until the build server processes your solution. Especially given that DB solutions tend to take longer than let’s say C# projects. You can of course re-build right in your VS studio, but that means a lot of thumb twiddling or lots of coffee runs.

One simple answer is to use msbuild.exe to re-build your DB “from outside”. I’ve been using a poweshell script to do this (surprise) with a lot of success. It gives me instant visual ques and provides a log file that I can use for further troubleshooting if needed. and also I can use the output for deploying the database to an actual SQL server using vsdbcmd.exe. After having read a great blog post on this subject by jamie thomson aka SSIS Junkie I felt compelled to share my build script. Here it is:

#MSBUILD.PS1
# automate build

[string]$output=""

$logpath="c:\PowerShell\log\buildLog.txt"

if (Test-Path $logpath)
{Remove-Item $logpath}

#output path
$buildOutput="C:\PowerShell\output"

$dbnames=@()

$msbuild="C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\msbuild.exe"

 $cdt=Get-Date

 #save output to a file
 $output=$output+ "script executed at " + $cdt + "`r`n"

 #also display the same interactively
 Write-Host "script executed at "  $cdt

 $output=$output +"`r`n"
 Write-Host

#load project list
 dir . -r  -include *.dbproj |where {$_.Name -notmatch "server"} |select-object fullname,name |
 % {

 $path=$_.fullname
 $dbnames +=$_.Name.replace(".dbproj","")
 $buildOutput=$buildOutput+"\"+$_.Name.replace(".dbproj","")

#for each dbproj file display its name
 $output=$output+ "********** building " + $_.fullname + " ********************" +"`r`n"
 write-host "********** building "  $_.fullname  " ********************"
 $output=$output+"`r`n"
 Write-Host

#execute the build and save output
 [string]$output1= &$msbuild $path /nologo /verbosity:n /p:OutputPath=$buildOutput

 $output=$output+$output1 +"`r`n"
 $output=$output+"`r`n"
 $output=$output+ "********** Finished building " + $_.Name + " ********************" +"`r`n"

 #check if the build succeeded
 if ($output1.contains("Build succeeded"))
 {
 Write-Host "Build Succeeded"

#display any warnings
$output1 | select-string -pattern "\d+?\sWarning\(s\).+$" |%{$_.Matches[0].Value}
 }

 #maybe it failed?
 if ($output1.contains("Build FAILED"))
 {
 Write-Host "Build Failed" -foregroundcolor red

 #display build warnings
 $output1 | select-string -pattern "\d+?\sWarning\(s\).+$" |%{$_.Matches[0].Value}
 }
 Write-Host
 Write-Host "********** Finished building "  $_.Name  " ********************"
 $output=$output+"`r`n"
 $output1=""
 $buildOutput="C:\PowerShell\output"
}

 # save output
 Out-File -filepath $logpath -InputObject $output

to download MSBUILD.ps1 as a file.

SQL server powershell war chest – connection provider

It is clear by now that MS SQL server and powershell are going to be good friends for a long time to come. So a prudent DBA should be building a war chest of script modules that can be reused in different cases. I will go through a series of short posts that will make every DBAs life easier and more productive.

let’s start with basics – connection provider. If you’re going to interface with a SQL server, you need to connect to it. Here is a very simple script that will do the job:

#get_conection.ps1

param(
      [string] $datasource="myserveraddress",
	  [string] $database = "MyDb",
	  [string] $authType = "Integrated",
	  [string] $username ="",
	  [string] $password=""
     )

## Prepare the authentication information.
if ($authType -eq "Integrated")
{
$authentication = "Integrated Security=SSPI;"
}
else
{ $authentication = ("uid={0};pwd={1};" -f $username,$password)	}

# Create the actual connection string
$connectionString = "Persist Security Info=True;Initial Catalog=$database;Data Source=$dataSource;$authentication"

## Connect to the data source and open it
$connection = new-object System.Data.SqlClient.SqlConnection $connectionString
$connection

As you can see, there is practically nothing to it. you can easily switch between SQL [regular] and Windows [integrated] modes of authentication. We’ll be using this script a lot in the future.

Download Get_connection.ps1

Working with suppress warnings in Visual Studio Database Solutions

MS Database Projects have come a long way from their humble beginnings especially with GDR and now full integration into VS2010, but plenty of issues remain. One of the more persistent problems is tracking references. Sometimes we just have to repress those warnings (code 4151). (http://msdn.microsoft.com/en-us/library/aa980442(v=vs.80).aspx) :

setting Suppress Warnings 4151
 

 

that’s all fine and good and gets us out of a bind short term, but if you really are going to use your DB project as intended, you should keep the number of objects that need this to a minimum. One problem is that once you set it, there is no visual way to identify a “suppressed” object. One way out of it is to parse the dbproj file. With Powershell, of course, it doesn’t get any simpler. Here is a script that I use. I’ve added comments for clarity:

#H4151.ps1

$root="."
#check if the output file exists, if so delete it
IF (Test-Path c:\temp\output.csv)
{Remove-Item c:\temp\output.csv}

# Create a file header
$output="DBProject,SQLObjects `r`n"

#get the files
$files=(dir $root -r | where {$_.extension -match "dbproj"}|Get-Item)
foreach ($file in $files)
{
#get the file naame
$ProjectName=$file.Name
$ProjectName

#get the second ItempGroup block
$fileString=Get-Content $file
$text_list=[regex]::Matches($fileString,"<ItemGroup>(.*?)</ItemGroup>") | Select-String "SuppressWarnings" | %{$_.line}

#let's add a root element to the list
$text_list="<root>"+$text_list+"</root>"

$xd=1$text_list

#get all the object Names
$xd.SelectNodes("/root/ItemGroup/Build[SuppressWarnings=4151]") | %{
$objName=($_.Include).split("\")[-1]
write-host $ProjectName,$objName;
$output=$output + $ProjectName + ","+ $objName+ "`r`n"
}

}

Out-File -filepath c:\temp\output.csv -InputObject $output -Append

Download a sample powershell script.