Wade Wegner

One of the advantages to the approach our teams building the Windows Azure Platform have taken is flexibility.  Recently, when I spoke at the Day of Cloud presentation, I recall Don Schwarz from Google making these two points (you can see video of his talk here):

• You can’t spin up your own threads in Google App Engine.
• You build your applications according to how Google thinks your apps should be built (the argument being that Google knows how to run highly available services at scale, which I think is a fair statement).

Now to be fair, there are good reasons for this – the Google App Engine has a number of very good use cases (Don Schwarz demonstrated one of them when showing the audience a multiplayer game running on Google App Engine).

Note: I’d like state for the record that I mean no criticism of other cloud vendors (i.e. Amazon, Google, and SalesForce).  I think each of them have a place in the market, and exhibit various strengths.  That said, I do believe that the Windows Azure Platform stands out as the only real platform that can bridge the chasm between cloud services and on-premises software. (Note to self: back this statement up in a future blog post.)

I would argue, however, that most enterprise developers require a little more flexibility when building out enterprise class applications.  The Windows Azure Platform provides this flexibility (I mean, come on – sometimes you just want to execute some native code!).

I decided to test how far I could take this flexibility in Windows Azure.

I am a big fan of Worker Roles in Windows Azure.  I really like the idea of having an asynchronous, “headless” service that’s working for me in the background.  I wanted to see if I could spin up an equivalent to a Worker Role in a Web Role, so that in addition to having my Web application running in a Web Role I could also get a service running in the background.

Why would I want to do this?  Well, here are a few of reasons:

• Asynchronous logging service running in every Azure instance (whether a web or worker role); this service might take information from the event log and write it to an Azure table.
• Clean-up operations (i.e. temporary scratch writes, etc.)
• Opening and managing socket connections to the service bus (this is my favorite scenario).

The list could go on.  Hopefully you get the point.

So, how would you do this?  Again, given the flexibility of the platform, there are many ways you can do this.  Below you’ll find one simple way.

1. Create a new cloud services project, and add a web role.

2. Create a new class and inherit from Microsoft.ServiceHosting.ServiceRuntime.RoleEntryPoint.  You’ll have to override the Start and RoleStatus methods.

using Microsoft.ServiceHosting.ServiceRuntime;

using System.Threading;


namespace Web

{

    public class MyWebWorkerRole : RoleEntryPoint

    {

        public override void Start()

        {

            while (true)

            {

                RoleManager.WriteToLog("Information", "The worker role is running.");


                Thread.Sleep(5000);

            }

        }


        public override RoleStatus GetHealthStatus()

        {

            return RoleStatus.Healthy;

        }

    }

}

3. Create a Global.asax page.

4. Create a method where you initialize and start and initialize your class that inherits from RoleEntryPoint.

private void StartWorkerRole()

{

    MyWebWorkerRole myWebWorkerRole = new MyWebWorkerRole();

    myWebWorkerRole.Initialize();

    myWebWorkerRole.Start();

}

5. Create local threading variables that you will use to run your method.

public class Global : System.Web.HttpApplication

{

    System.Threading.ThreadStart ts;

    System.Threading.Thread t;


    ...

5. Update the Application_BeginRequest method to invoke your method on a new thread.  Be sure and check to see if the thread is already running, otherwise it will get started multiple times.

protected void Application_BeginRequest(object sender, EventArgs e)

{

    if (t == null)

    {

        ts = new System.Threading.ThreadStart(StartWorkerRole);

        t = new System.Threading.Thread(ts);

        t.Start();

    }

}

And that should do it.  Now you have a class that will run asynchronously in all your Web Role instances.  And since it inherits from the RoleEntryPoint, you can leverage this technique for both your Web and Worker Role instances.  I’d recommend placing this class in a separate class library and adding a reference to it from your various projects.

Note: It is highly probable that there will be some changes to the APIs and assemblies come PDC.  While these concepts will stay valid, the underlying code will probably change.  If this happens, I’ll be sure and provide an update.

I hope this helps!

Note: This is a guest post from George Huey, Architect Evangelist in the Developer and Platform Evangelism group.

When you upload your data into SQL Azure, SQL Azure replicates your data to three different locations in order to provide triple redundancy.  Therefore, it needs a little more time to get the data in the proper places.

One of the things that we found out during a series of Windows Azure Platform Migration Labs held in the Chicago MTC is that you cannot upload hundreds of thousands of records without giving SQL Azure time to catch up.  Consequently, you have to chunk your data and give SQL Azure time to process each chunk before uploading the next chunk of data.

The tool that we used for migrating our customer databases to SQL Azure was the SQL Azure Migration Wizard.  The migration wizard uses BCP to download data from an on premise SQL Server database and then uses BCP to upload the data to SQL Azure.  BCP allows you to specify the first row (-F), the last row (-L), and the batch size (-b).  These options will allow you to chunk the data beginning uploaded to SQL Azure.  For example:

BCP MyDb.dbo.Transactions out Transactions.dat -E -q -n –T 

The above command extracts data from table Transactions in the database MyDb.  At the end of the BCP output, you will find the number of records copied to file (for example: 2,524,520 rows copied).

In order to upload in chunks, you would do something like this:

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F1 –L250000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F250001 –L500000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F500001 –L750000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F750001 –L1000000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F1000001 –L1250000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F1250001 –L1500000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F1500001 –L1750000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F1750001 –L2000000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F2250001 –L2500000 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

BCP  MyDb.dbo.Transactions in  Transactions.dat -E -q -n –b 5000 –F2500001 –L2524520 -S  tcp:azureserver.ctp.database.windows.net -U admin@azureserver -P password

Note, you will have to put some kind of delay between BCP commands to give SQL Azure time to store the data (say 15 seconds).  You will probably find that sometimes the 15 seconds is not enough time and that, during the upload of one of your BCP chunks, SQL Azure might shut it down.  If that happens you will see something like this happen:

5000 rows sent to SQL Server. Total sent: 145000

5000 rows sent to SQL Server. Total sent: 150000

5000 rows sent to SQL Server. Total sent: 155000

SQLState = S1000, NativeError = 21

Error = [Microsoft][SQL Server Native Client 10.0][SQL Server]Warning: Fatal error 40501 occurred at Oct 30 2009  4:15PM. Note the error and time, and contact your system administrator.

BCP copy in failed 

From the above BCP output, you will see that a total of 155,000 rows were sent before SQL Azure closed the connection.  Thus you would have to adjust your next BCP command to start at your –F value + 155000.

While this process works reasonably well, it can make the process of uploading data a little tedious if you have a large number of tables with a large number of records per table.  In order to simplify the process, we have modified the SQL Azure Migration Wizard to do all of this work for you.  It allows you to specify the chunk size, the batch size, and the time to wait between BCP chunks in SQLAzureMW.exe.Config.  It also catch BCP errors and adjust for records processed and then retry.

Try it out, review the source code, and be sure to provide some good feedback!