Blog

Building SSDT SSRS projects (.rptproj) with a build server (TeamCity)

Microsoft .NET

To build SSDT projects with a build server you can use devenv.com or devenv.exe, but not MSBuild.exe because it is not supported by the SSDT tools for Visual Studio 2015. Unfortunately that means installing a licensed copy of Visual Studio on your server. For me this was necessary anyway because we also have to build various Microsoft Office Add-ins and Visual Studio Setup Installer projects (.vdproj).

In TeamCity there are several build runners available, the two common ones for building Visual Studio projects being the Visual Studio (SLN) runner and MSBuild. I had assumed that the SLN runner actually called devenv.exe or devenv.com, but that is not the case. The SLN runner is just more simple and utilizes your existing solution file to produce an in-memory MSBuild file on the fly, while MSBuild is used for more advanced scenarios such as also using MSBuild Tasks as part of your build process. Thus, the SLN runner actually just calls MSBuild.

For building SSDT projects with a build server, you must actually use a command line build by invoking devenv.com. The devenv.com process is similar to devenv.exe, but is designed specifically for command line usage. You can do the same with both, but devenv.exe has the possibility to hang your builds because it may become interactive on the server, waiting for user input, so this is not recommended. Configuring the build step appropriately is really straight forward.

ssdtteamcitybuildstep

The path to denvenv.com will vary depending on your version of Visual Studio, but will be located in C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.com. Just substitute your version in place of 14.0 and you’re set. You can also build your solution with various other parameters that are supported (run devenv.com /? from the command line), such as /Build and with different platforms /Build "Debug|x86".

MSBuild Support for SSDT

SQL Server Data Tools, or SSDT, has been the replacement for Business Intelligence Studio for some time now. The first major release of the tools usually comes shortly after the initial release of a new major version of Visual Studio, and it wasn’t until June this year that the tools made their debut for Visual Studio 2015. The release cycles are getting better as it appears in the next edition of Visual Studio they have already been working on the tools, hopefully for a release that is in sync with the main IDE debut. The tools we already have for Visual Studio 2015 do not currently support building the various SSDT projects via MSBuild, but it appears that they have introduced this functionality, albeit limited this far, in the Visual Studio “15” Preview 4 as a Nuget package. From a reply to a comment I posted, it appears it only supports building SQL Server Database projects (.sqlproj) at this time.

Threads, locks, deadlocks, race conditions, and thread safety in multi-threaded code

Microsoft .NET

Before proceeding, I am going to state that I will talk about multi-threading with a user interface in a different blog. It warrants its own post, so here we will focus on pure threading concepts.

At some point in your career you will need to write a multi-threaded application, but you probably won’t do it right. Even worse, you won’t know it, and arguably you can get away with it most of the time. Some developers go their entire careers without touching multi-threading, which I have found to be more common in web development than client applications, services, or systems development. I believe it is something every programmer should understand, just like I believe every programmer should understand pointers.

My definition of a thread is a unit of work that can be scheduled for execution. A process in Windows can contain one or more threads where work is scheduled to be processed by the CPU. In processors with only one core, true concurrency doesn’t exist where two units of work can occur asynchronously. Although it appears that is the case, otherwise your system would seem unusable, a CPU is able to switch between threads fast enough that it is capable of giving each thread a fair slice of CPU time resulting in a responsive system. Processors with faster clock speeds will result in faster context switching, and ultimately, the ability to process more threads in less time. True concurrency exists on systems with processors that have more than one core, where each core is capable of processing threads independently from the other cores. Of course this all is still managed by a thread pool or scheduler which helps prioritize which threads will get CPU time and when, and their priority.

Now that you understand a bit about what a thread is, why use them? You’re using them whether you know it or not, every application already has them before you even write a line of code in your IDE, such as a thread required for a console window. Usually threads are used for long running tasks as to not block and prevent another operation from occurring. This is very common in a UI, such as Internet Explorer where you want to download a file on a separate thread so you can keep browsing the web. If this was done on the same thread, you would be blocked from doing anything until the download completes, either in full or due to cancellation. If that was the case, you wouldn’t even be able to cancel the download because the UI would be blocked as well. Other considerations might be performance to split large amounts of work up if processing power is available, or to utilize more than one core (parallel programming).

Lets take a look on how you would create a thread in C#.

static void Main() 
{
    Thread thread = new Thread(DoWork);
    thread.Start();
}

static void DoWork() 
{
}

This is simple, requiring us to only create the thread and invoke Start() to begin the unit of work that will be performed by DoWork(). You can observe the number of threads in an application via Windows Task Manager. You will need to enable the threads column though; it isn’t enabled by default.

Threads

You would need a simple application that would allow you to observe this. Sometimes the thread count will be adjusted by the operating system. You need to remember that threads are already there for bare bones things the operating system needs to work, such as your console window. At the end of the day though, you should be able to observe the thread count increase using the sample below. Another note to add is you will notice that even though we are kicking off a seperate thread, the console application doesn’t exit, and we will talk about that later in the article.

static void Main()
{
    Console.ReadKey();

    Thread thread = new Thread(DoWork);
    thread.Start();

    Console.ReadKey();
}

static void DoWork()
{
    while (true) ;
}

I’ll give one more example, but I think by this point the concepts should resonate. Lets look at a simple program that contains two threads for processing work.

class Program
{
    static int value;

    static void Main(string[] args)
    {
        Thread thread1 = new Thread(DoWork1),
               thread2 = new Thread(DoWork2);
            
        thread1.Start();
        thread2.Start();
    }

    static void DoWork1()
    {
        while (true) Debug.Print((value++).ToString());
    }

    static void DoWork2()
    {
        while (true) Debug.Print((value -= 2).ToString());
    }
}

When you run this multiple times and compare the output of each, you should immediately notice that the output varies. Here are two runs that I performed.

run one -1, 0, -3, -2, -3, -2, -4, -1, -6, -5, -4, -3
run two -1, 0, -3, -3, -2, -3, -3, -4, -3, -4, -3, -5

Understanding how multi-threading works this is understandable. Each thread gets a slice of CPU time to execute, but there is no guarantee what that exact slice will be or when it will be. All you know is that there is a guarantee of getting a slice of CPU time. Your threads are also competing with all other threads in the operating system, and by this point you should be visualizing and connecting the dots. A user complains that your software is slow and you find out that they are running another program that is consuming 90% of the available CPU, you know your threads are not getting the time slices they need or in the time that you need them.

Background and Foreground Threads

Look closely at the example program. You will notice that there is no blocking call after invoking the Thread.Start. If you caught on, you are wondering why the console application does not exit immediately since a thread is asynchronous and the main method should be returning and ending execution. The Thread class has a property called IsBackground, and this is important for you to understand. Lets look at the Msdn documentation.

A thread is either a background thread or a foreground thread. Background threads are identical to foreground threads, except that background threads do not prevent a process from terminating. Once all foreground threads belonging to a process have terminated, the common language runtime ends the process. Any remaining background threads are stopped and do not complete.

A thread by default is a foreground thread. While Main has completed its work, the runtime prevents the process from exiting until the threads have completed their work. In this case, that is not until you terminate the application because they are in an infinite while loop. You can change this behavior simply by setting IsBackground to true, which will allow the program to exit without requiring threads to complete. By doing this, the console application would exit immediately and you would need to add a blocking call, such as Console.ReadKey to allow the threads to run.

Lock

You understand by now how to create, work with, and how threads work, but what about a lock? You have probably read about it and even seen a code sample, but you have probably often wondered what I did when I originally read about a lock, which I’ll describe in a few questions.

  • What is a lock?
  • When is a lock appropriate to use?
  • How do I use a lock correctly?

Lets look at what a lock is defined by the Msdn documentation first.

The lock keyword marks a statement block as a critical section by obtaining the mutual-exclusion lock for a given object, executing a statement, and then releasing the lock.

Now lets turn that into something more understandable.

ThreadLock1

The image above is simple. You have two people but there is only one key to their only car. If person one takes the key to the car to go run an errand, person two has to wait until they come back and make that key available again before they can drive the car for their own errand. This has both a advantage and disadvantage. The advantage is that having only one key ensures that we don’t try to use the car at the same time. That seems like a silly scenario, but imagine both people running to the car and starting to fight over who gets to use it, and the argument lasts several hours. Basically, several hours were lost, and nothing was accomplished. It was a deadlock. This is the advantage of a lock, because it prevents this sort of scenario from happening by acting as a mediator and forcing each thread to wait their turn to use the resource.

The disadvantage to this is that the thread who has the mutually-exclusive lock may never complete and the lock could last forever. This is also a deadlock, and is the worst sort because the other thread will be waiting around until the lock is released, which might be never. These are the kinds of situations that can hang an application making you scratch your head.

Race Condition

Before I can show you how to use a lock or demonstrate a deadlock, we need to demonstrate a race condition which is the root problem you will solve with a lock in the first place.

class Program
{
    private static List<Guid> guids = new List<Guid>();

    static void Main(string[] args)
    {
        Thread thread1 = new Thread(DoWork1),
               thread2 = new Thread(DoWork2);
            
        thread1.Start();
        thread2.Start();

        while (true)
        {
            Console.Clear();
            Console.Write(guids.Count);
        }
    }

    static void DoWork1()
    {
        while (true) guids.Add(Guid.NewGuid());
    }

    static void DoWork2()
    {
        while (true) guids.Add(Guid.NewGuid());
    }
}

This code will most likely result in an ArgumentOutOfRangeException. I have to say most likely because you might be, in terms of probability, 0.01% lucky enough to run the sample through without observing a race condition. It technically could happen, but the chance is extremely rare, but I have to be truthful from an educational standpoint that it is remotely possible.

Moving on, this results in a exception because List<T> is not thread-safe. Thread safety is a phrase used to describe a mechanism, component, or piece of code that was created with multi-threading in mind, and is guaranteed to work with multi-threaded code when accessed from one or more threads concurrently. Authors of thread-safe code are responsible for ensuring thread-safety, and the definition of expectations of thread-safety will vary which is where you will have to rely on documentation, if there is any. Microsoft is great about making sure they document thread-safety throughout .NET, but this may not be the case for third party code.

The threads in the sample above are accessing the list as fast as they possibly can. When you add or remove an item from a list it will automatically increase its capacity if necessary. What is happening here is the first time it needs to increase capacity, it has been accessed by the other thread which has added an item, resulting in a new capacity less than the current size. The exception message in this case is extremely articulate about the problem.

We’re going to solve this problem with a lock.

class Program
{
    private static List<Guid> guids = new List<Guid>();
    private static object key = new object();

    static void Main(string[] args)
    {
        Thread thread1 = new Thread(DoWork1),
               thread2 = new Thread(DoWork2);
            
        thread1.Start();
        thread2.Start();

        while (true)
        {
            Console.Clear();
            Console.Write(guids.Count);
        }
    }

    static void DoWork1()
    {
        while (true) 
            lock (key)
                guids.Add(Guid.NewGuid());
    }

    static void DoWork2()
    {
        while (true)
            lock (key) 
                guids.Add(Guid.NewGuid());
    }
}

Make sure you read the Msdn documentation on lock, I’m not going to repeat that here, but it is very important that you use an object that is private and not accessible by anything beyond your control. The lock code itself is simple and you should be able to correlate it with the picture of the people, key, and car. What you need to take from this is that key is just an object in memory that acts as the mediator that forces the threads to wait their turn before they can run their critical section of code. Re-run the sample and you may verify that the race condition is resolved.

Deadlocks

Locks are simple and as promised we can demonstrate a deadlock.

class Program
{
    private static List<Guid> guids = new List<Guid>();
    private static object key = new object();

    static void Main(string[] args)
    {
        Thread thread1 = new Thread(DoWork1),
               thread2 = new Thread(DoWork2);
            
        thread1.Start();
        thread2.Start();

        while (true)
        {
            Console.Clear();
            Console.Write(guids.Count);
        }
    }

    static void DoWork1()
    {
        while (true) 
            lock (key)
                while (true)
                    guids.Add(Guid.NewGuid());
    }

    static void DoWork2()
    {
        while (true)
            lock (key) 
                guids.Add(Guid.NewGuid());
    }
}

thread2 might get a chance to run, but when thread1 acquires a lock it will deadlock thread2 causing it to wait forever. This is because we added a infinite loop inside the lock in thread1 so it will never be released. This is a very simple demonstration of a deadlock, but the important thing to note here is a deadlock can be a single thread or many threads could be deadlocked waiting on one another; it just depends on your code.

Worthy mention: Task Parallel Library

I have written other articles on the TPL, but what you need to know is it is the modern way to start tasks that run as threads on the managed thread pool (warrants its own article). Everything I have shown you has been using System.Threading.Thread which has been around forever and a day. Let me say this now, there is nothing wrong with using Thread over Task, but the api surface that the TPL has provided has made writing multi-threaded code much easier.

I’m not going to write much more about TPL in this article, but I want to end the article leaving you with the understanding of how multi-threading works, because as you begin to use TPL, all of these concepts still apply and it’s just another layer of abstraction.

Conclusion

There is still a lot I did not cover about multi-threading. How do I know when a thread completes? How do I share thread state? What is a thread pool? What is thread synchronization? I like to say to colleagues, everything in software development is its own college degree. You can spend years learning about multi-threading all the way down to the operating systems inner workings. I recommend you do, but I hope at least this article covered some areas of multi-threading that I always see a lot of repeated questions for, and generally in my experience a lot of developers know nothing about, struggle with at first, or never fully understand it and just move on.

Happy coding!

Adding support for reading 3 of 9 Extended barcodes (containing lowercase characters) with GdPicture9

Microsoft .NET

3 of 9 barcodes are just about everywhere, and are one of the easiest barcodes to work with being widely supported by many SDK’s. 3 of 9 originally supported only a subset of the ASCII character set, and later was extended to support the full range of characters. Often you will find 3 of 9 barcodes in the form of a font style that you can use in word processors, web applications, and other types of software. They can be copy and pasted. They are easy to draw, and most importantly, they are very easy to understand. You can read more on Wikipedia about it, but lets talk more about how GdPicture SDK deals with it.

code39-sample

GdPicture does support drawing 3 of 9 barcodes with the extended character set using the Barcode1DWriterType.Barcode1DWriterCode39Extended enumeration with Barcode1DWrite, but I found that it lacked the ability to read them using Barcode1DReaderDoScan. If you look deeper into the enumerations available for these methods, oddly enough both Barcode1DWriterType.Barcode1DWriterCode39 and Barcode1DWriterType.Barcode1DWriterCode39Extended exist for drawing the barcode, but only Barcode1DReaderType.Barcode1DWriterCode39 exists for reading when using Barcode1DReaderDoScan.

gI_78908_LOGO-GD9

Now, let me clarify before we look at a solution. GdPicture does technically read the barcode with extended characters, but it doesn’t actually return the extended ASCII characters. Instead, it returns each uppercase character prefixed with the + character, which is a token indicating the character casing, such as the sign in a signed integer. In the examples below we will look at the most common scenario which is support for lower case characters, which are only part of the extended character set.

string threeOfNine    = "ABC123"; // normal character set
string threeOfNineExt = "abc123"; // extended character set

These are the raw values you would normally expect to see when reading 3 of 9 barcodes using a proper SDK. In GdPicture however, let us take a look at the difference when reading the extended barcodes.

string threeOfNine    = "ABC123";    // normal character set
string threeOfNineExt = "+A+B+C123"; // extended character set

This pattern is very consistent, so we can write a simple function to parse and return the correct character casing.

/// <devdoc>
/// This method is for GdPicture9's lack of support for *reading* the 3 of 9 Extended character set,
/// which allows for lowercase characters. GdPicture will still return a barcode value, but each lower
/// case character is preceded by + character to indicate the casing difference.
/// 
/// ie. TEst (raw value of *TEst*) would be returned as TE+S+T (raw value of *TE+S+T*)
/// </devdoc>
private static string Get3Of9ExtendedBarcodeValue(string s)
{
    var newValue = string.Empty;

    for (int i = 0; i < s.Length; i++)
        if (s[i] == '+')
            newValue += s[++i].ToString(CultureInfo.InvariantCulture).ToLower();
        else
            newValue += s[i];

    return newValue.Replace("+", string.Empty);
}

The method is pretty straight forward, iterating over each character and identifying the token. When the token is found we do a forward lookup in the array, which will be the actual character value, and we know to return the lowercase value. We also increment the loop counter at the same time since we are processing that character. Otherwise if no token is found, we simply return the character as is.

Simple!

Now, I have only written this function to account for the casing differences between the character sets because that was the scenario I needed to solve for the project I was working on. I could have used other barcode libraries, but we already had licensing for GdPicture and it is significantly faster than most of the libraries out there, and I tested quite a few. I might revisit this blog at a later time and write a new one discussing the other extended characters, but for now we will leave it there.

Happy coding!

Custom Method Names via Attribute Routing in ASP.NET Web API 2

Microsoft .NET

I am using Web API 2 in my ASP.NET MVC 5 project, and I found myself needing custom routes very quickly. In my scenario I’m building a single-page application using AngularJS, so all the data is received and sent to and from web api’s from ajax calls.

In my scenario I have a collection projects, and each project has a collection of releases; ultimately my web api routing and controller looked like the following.

config.Routes.MapHttpRoute("DefaultApi", "api/{controller}/{id}", new { id = RouteParameter.Optional });
config.Routes.MapHttpRoute("ActionApi", "api/{controller}/{action}/{id}", new { id = RouteParameter.Optional });
[RoutePrefix("api/Projects")]
public class ProjectsController : ApiController
{
    public ProjectDTO[] Get()
    {
        ...
    }

    public ProjectDTO Get(int id)
    {
        ...
    }

    public HttpResponseMessage Post(Project project)
    {
        ...
    }

    public void Put(Project project)
    {
        ...
    }

    public void Delete(int id)
    {
        ...
    }
}

This is the default, or at least, most common method of using web api based on the HTTP verbs used when calling the service. While this is great for the most basic CRUD operations, I needed to facilitate relational data and have custom named methods. There are many ways to accomplish custom method names gracefully, each has their own pros and cons, but ultimately I’m just going to cut to the chase and demonstrate attribute routing, which I found to be the easiest, and most useful. I spent several days researching ways to do this, and no matter what I did I ran into some limitation, funky issue, or an approach that required custom routes or additional code, or even uninstalling WebDAV or re-configuring your web.config file – sigh. To say the least, I felt they were all terrible solutions.

Finally, I came across this article.
http://www.asp.net/web-api/overview/web-api-routing-and-actions/attribute-routing-in-web-api-2

In short, it saved my sanity. I was able to add a web api as simple as this.

[HttpGet]
[Route("{id}/Releases")]
public ReleaseDTO[] Releases(int id)
{
    ...
}

No changes to my routing. No changes to my web.config file. No uninstalling WebDAV. No weird limitations. What this also gave me was the control over the routes that I wanted so I could have a consistent url scheme that made sense, since the data is relational. The url scheme would be ‘/api/projects/{projectId}/releases’, which makes complete hierarchical sense.

After all the research I did, this would be the approach I would recommend when you need to expand your web api past the basic HTTP verb support. I tossed the idea of using a lot of separate controllers, but felt it would be a maintenance nightmare, and a bit silly to have too many, albeit small controllers when I could logically group things a bit better.

Learning MVVM from the ground up for building real-world software

My two MVVM video series` on YouTube have become very popular, and I have every intention of continuing getting more information out there on MVVM through my Enterprise series. If you haven’t had a chance to check out the videos, here is the full list below.

MVVM Session 1 – http://youtu.be/EpGvqVtSYjs
MVVM Session 2 – http://youtu.be/Dzv8CtUCchY
MVVM Session 3 – http://youtu.be/OOHDie8BdGI

Enterprise MVVM Session 1 – http://youtu.be/lool8Ut58Xw
Enterprise MVVM Session 2 – http://youtu.be/CpVNMmefvDo
Enterprise MVVM Session 3 – http://youtu.be/RbfqNOSyYok
Enterprise MVVM Session 4 – http://youtu.be/7SwgqLJCLI8

MVVM Questions and Answers – http://youtu.be/j8Fke2cUTTc

I want to thank everyone who has followed my series and provided feedback. It has emboldened me to ensure I provide quality content and continue the effort of producing videos. MVVM in of itself is a simple concept, but a lot of questions stem from people trying to build real-world software, and finding material for that can be difficult. From the basic stock ticker demos to simple CRUD applications and Msdn articles, none of these really actually demonstrate a full stack MVVM application using dependency injection, service locater’s, messengers, unit testing, mocking; basically the full stack. This is what I intend to do through the enterprise MVVM series. Maybe I will write a book at some point. Who knows. Right now though, I’m enjoying producing the video content.

I also just ordered new sound equipement, the RODE PodCaster microphone, as my Blue Yeti one sort of, well, fizzled out I guess. Bummer, but looking forward to the new one already. Anyhoot, check out the videos if you haven’t already. And again, thank you to my followers!!

Enterprise MVVM in WPF: Adding Data and Business Layers using Entity Framework

Second video in the Enterprise MVVM in WPF series where I demonstrate the practice of adding a data layer and business layer using Entity Framework and test driven development (TDD). Learn how to make a unit of work business context that you can use to encapsulate your business rules and logic, but still be able to go a layer down to entity framework.

Grab the source code here, and the slide deck here. Don’t forget to like and subscribe if these videos helped you. 🙂

Series:
Part 1: https://www.youtube.com/watch?v=lool8Ut58Xw&hd=1
Part 2 (this video): https://www.youtube.com/watch?v=CpVNMmefvDo&hd=1

Enterprise MVVM in WPF: ViewModel Validation using Data Annotations

I finally had some time off from my busy work life to produce a few WPF MVVM videos on my YouTube channel. It is a bit of a relief to finally get some content up for you guys again. Prior, I believe the last video I had done was a bit over a year ago. The video is the irst video in a new series on what I consider to be enterprise MVVM development for WPF. We will begin covering practical application development scenarios that you will encounter in the real-world, and focus on building high quality, low maintenance, enterprise solutions.

Video:

You can grab the demo code here, and the slide deck here. Like and Subscribe for more videos!

Series:
Part 1 (this video): https://www.youtube.com/watch?v=lool8Ut58Xw&hd=1
Part 2 :https://www.youtube.com/watch?v=CpVNMmefvDo&hd=1

Video Notes:
– I apologize for the low audio, the new microphone (Blue Yeti) picks up pretty much all background noise, and my desktop PC is very loud. I will be looking into ways to resolve this, and if anyone has recommendations please let me know.
– When I originally created the projects I had Visual Studio set to .NET 4. If you are using Visual Studio 2012 or 2013, make sure you set your projects to target .NET 4.5 or .NET 4.5.1.
– There are two parts where the audio may seem like it was edited, and this was true. Sometimes noise from other people makes it into a video and I do my best to edit it out without ruining the video content.
– The downloadable source code will be much cleaner than what was demonstrated in the video. I will be focusing more on the approaches in my videos, and cleanup can always come later.
– The downloadable source code will also contain many more unit tests, including some unit tests for the view models that we were not able to do in the screencast originally.

Thank you everyone for all your support! I have been listening to the questions and feedback in the comments, and paying attention so that I will be focusing my attention on the questions and topics you guys are asking for.

Introduction to Test Driven Development in .NET

Testing has always interested me to some degree. Not from the perspective of manual testing or quality assurance, but from the perspective of the developer behind the desk who just wrote the code. How can a developer, who is accountable for their own code, ensure that they are producing the highest quality code with a low defect count and high maintainability rating? One answer for many years now has been the development process of TDD, or Test Driven Development.

TDD is not just about unit tests; it’s actually an entire thought process with a set of core fundamentals you should follow, and it really changes the way you think about your code, and eventually the way you write it.

I did a thirty minute screencast on TDD in an introductory manner. I go through the steps of designing and building a simplified inversion of control container through the process of TDD.

Video
Watch on YouTube now.

Source Code download:
Download from my Skydrive

Powerpoint Slides
Download from my Skydrive