Create a Custom Object with .NET using the Force.com Metadata API ... and No Proxy Classes!


Update 2/24/14

Over the weekend I took much of what I wrote below and packaged it up in a NuGet package. You can use the NuGet to accomplish everything I wrote below with fewer lines of code. Full details are available on the WadeWegner.Salesforce.SOAPHelpers Github repository and you can get the NuGet package with the following command:

    Install-Package WadeWegner.Salesforce.SOAPHelpers

Enjoy!


Recently I've written a lot about using the Force.com REST API with .NET. I've also created a Force.com Toolkit for .NET, a Chatter Toolkit for .NET, and a set of sample applications that make it easier for developers to interact with these APIs. Using the REST API you can write custom integrations with your applications in Salesforce, interacting with all of your metadata through simple HTTP calls using JSON.

There are, however, a few limits to what the REST API can accomplish. In particular, one of the requests and/or questions I see over and over is how to create custom objects and fields using the APIs. If you look at the Developer Forum you’ll see lots of people (especially .NET developers) ask these questions.

The answer is relatively simple: use the Metadata API.

Of course, the reality is that there’s more to it. In particular, if you’ve only used the REST APIs (or it’s been years since you’ve had to use SOAP as was the case with yours truly), this may present a challenge as the Metadata API is SOAP-based.

But not to fear! The utility of these APIs is worth the few extra minutes and lines of code.

When starting with these SOAP APIs, most developers will typically download the WSDL files, generate class proxies, and code against them. This is a fine approach and there’s a healthy amount of examples that show how this works.

That said, generating full class proxies against the WSDLs can be a bit heavy-handed when we want to simply do a few basic operations, like create a custom object with some fields. Furthermore, if you’re like me, you may desire more insight into what’s happening in the code and the HTTP requests, and don’t want to rely on all the code that’s automatically generated for me. Personally, I have no problem writing a little more code if it means I have more control and insight.

In this post, I plan to discuss the following through narrative, raw HTTP requests/responses examples, and (of course) code:

  1. Demystify the Enterprise and Partner APIs.
  2. Explain and demonstrate the SOAP login process.
  3. Explain and demonstrate how to use the Metadata API to create a custom object and field.

All this without generating proxy classes using the WSDLs.

Ready to get started? Let’s first look at the Enterprise or Partner APIs.

SOAP API: Enterprise and Partner WSDLs

The SOAP API lets you create, retrieve, update, or delete records within Salesforce. There are over 20 different API calls that let you perform a number of important tasks and operations. In many ways, it’s similar to the REST APIs (except it’s SOAP ... sorry for the obvious).

As part of the SOAP API, there are two different web services you can use, referred to as Enterprise and Partner. They are functionality equivalent and share the same operations yet differ in how developers interact with them. Each allows you to generate WSDL file to simplify your interactions. Understanding the differences between these two web services and WSDLs is important.

The Enterprise WSDL is a strongly typed version of the API, meaning that an object in Force.com will have an equivalent object in Java, .NET, or whatever environment is accessing the API. The benefit of this model, where your code is compiled against a specific schema, is that coding is generally easier and more straightforward (as you don't need to deal with any underlying XML structures) and safer in the sense that data and schema dependencies are resolved at compile time, not at runtime. The downside of that strongly typed model is that it only works on a single Force.com account's schema, as it is bound to all of the unique objects and fields that exist in that account's data model. This means that when the metadata in your organization changes you’ll need to update your WSDL and regenerate your proxy classes.

What if you create a solution that you want to work across multiple accounts from different customers? In that case, you need a weakly typed representation of the data. This is is precisely what the Partner WSDL provides.

With the Partner WSDL, the developer is responsible for marshaling data in the correct object representation (which typically involves diving into the XML), but is freed from being dependent on any particular data model or Force.com account.

Put simply, if you are a company using the API exclusively on your own account, you'll typically want the Enterprise WSDL. If you are an ISV using the API across many accounts, you'll typically want the Partner WSDL.

Regardless of which path you use, you’ll have to use the SOAP API to authenticate and use the Metadata API. If this is your only goal, it doesn’t really matter if you use the Enterprise or Partner web service.

For a detailed account of the SOAP APIs please refer to the SOAP API developer documentation.

Authenticating with the SOAP API

To use the SOAP or Metadata APIs you need to login and start a client session. To do this, you must login and obtain a sessionId and and server URL. The sessionId is used in subsequent calls to the web services to identify your session and the server URL identifies the appropriate URI with which you make your subsequent requests.

Note: don’t forget to append your security token to your password if logging in from a location unknown to Salesforce. You will be blocked!

There are two different URIs for logging in, depending on whather you’re using the Enterprise or Partner web service.

  • Enterprise: https://login.salesforce.com/services/Soap/c/[version]/[organizationId]
  • Partner: https://login.salesforce.com/services/Soap/u/[version]/[organizationId]

Here’s an example of what a raw login request looks like for the Enterprise web service:

The request is nearly identical for the Partner webservice:

I’m sharing the raw requests in this post as I think it is profoundly more useful than just showing how to create the request in a single language. The reality is you can use any language that supports HTTP communication and, even if you do just need the .NET code, you’ll have a better understanding of the process if you know what the raw request looks like.

That said, how do we create these requests in code without using the proxy classes generated by the WSDLs? Let’s see.

I wanted to write a single method that worked against both the Enterprise and Partner API. Required? Absolutely not. If you just want to use one you can easily refactor the following code.

When you make the request you’ll get a response. Whether you use the Enterprise or Partner webservice you’ll get loginResponse that pretty much looks like this:

The only difference between the Enterprise and Partner web service is that the Enterprise web service will give a response with the following namespace ...

    xmlns="urn:enterprise.soap.sforce.com"

... whereas the Partner web service will give a response with this namespace ...

    xmlns="urn:partner.soap.sforce.com"

Again, let me reiterate that the loginResponse information you receive is identical, so which one you use doesn’t matter; there’s no material difference when it comes to logging in.

There’s a lot of information in these responses. For our purposes in this post, we really only need the sessionId and metadataServerUrl. As previously mentioned, the sessionId allows us to authenticate when making a call to the various SOAP APIs and the metadataServerUri is the URL we need to use when interacting with the metadata API during subsequent calls.

So, let’s look at what’s required in order to deserialize all this information into .NET objects and then grab the information.

First, we need to define these objects. This is a little tricky in .NET, as we need to attribute our classes with the XML namespace in order to appropriately deserialize. I took an approach of defining two classes that are nearly identical yet live in different namespaces.

If you choose an approach where you simply use either the Enterprise or Partner web service you can simply choose one or the other.

The code for serializing and returning the response is pretty straightforward.

At this point we now have all the information required in order to interact with the Metadata API and create our custom objects and fields.

Creating a Custom Object using the Metadata API

It sure has taken us a long time to get here. However, using the above, you now have the ability to easily authenticate to the SOAP APIs. Throw these methods into a library and save them - I’m sure you’ll use them over and over.

What does the SOAP look like for creating a custom object?

For SOAP this is pretty standard. We have a SOAP envelope and, within the body, we’re passing in the metadata used to create our custom object.

We’ll get the following response from the Metadata API:

One thing to point out is that this SOAP request only created the custom object. We don’t have any fields yet! We need to issue another request in order to create the fields.

Here’s how we add an additional field (you’ll see it’s unsurprisingly similar):

The surrounding SOAP envelope when creating a custom field is identical to the SOAP envelope used for creating a custom object. What changes is the metadata in the body.

So, what’s an efficient way to approach this in code? Here’s the approach I’ve taken.

Let’s first create a method that handles the surrounding SOAP envelope, makes the HTTP request, and then returns the raw result.

You can see that this approach lets us easily pass in the metadata required to either create an object or field, and will build our SOAP envelope appropriately.

From here, let’s create a method for creating the custom object metadata, call our Create method above, and then deserialize the response into an appropriate .NET object.

We can do the same for a custom field.

Both methods above are extremely similar. With a little more refactoring we might easily simplify this some more, but I’ll leave that to you. (You need something to work on, right?)

Now, given everything we’ve created above, how might we run this?

Pretty slick. As you can see, the only things you need to bring to this approach are your username, password, and organization id. The first two should be easy for you to get. How do you get your organization id? Easy. Click on Setup | Company Profile | Company Information, and in the right hand column towards the bottom you’ll find your Organization ID.

Conclusion

So, how can we confirm that this work?

Custom Object and Field

Log into your Salesforce organization. Under recent items you should see the custom object and fields you’ve created. Go ahead and click on the custom object and you can confirm that it has the custom field.

There you have it! A reasonably simple approach to establishing a session with the Enterprise and/or Partner APIs and using the Metadata API ... all without having to generate class proxies from the WSDL!

You can grab all the code on Github: https://github.com/wadewegner/CustomObjectsUsingSOAP

I hope you found this useful.

Send a Text Message to Your IoT Thermal Printer? No Problem. (And Print Your Lunch Menu Too!)

How do you send a text message from your phone and have it print on a thermal printer that's sitting on your kitchen counter?

iot-beer

Hang on, we'll get to that shortly!

Shortly after Dreamforce last year I was turned on to Adafruit Industries by my teammate Reid Carlberg. I've since built a number of cool little gadgets, including the Brain Machine and the TV B Gone. Both projects were a ton of fun and a great way to get started with soldering and simple circuitry. It's been great fun having the TV B Gone when I travel. There's nothing more satisfying than turning off loud, annoying TVs (invariably on CNN) at the airport when you're trying to read or work.

I don't recall exactly how/where I first heard of it, but I was immediately drawn to the Internet of Things Printer for Raspberry Pi. I think it's the combination of a number of things that got me interested:

  • A small thermal printer that doesn't require an ink cartridge.
  • It's powered by a Raspberry Pi, providing lots of opportunities for coding.
  • A miniature USB WiFi allowing it to run almost anywhere.

And of course, having three young kids, I thought of so many different ways the printer could be fun and useful (although, to be 100% honest, I haven't found all that much utility in it yet, but it sure has been fun).

The construction of the printer was fun. It took me a couple of days, but I'm slow.

Simply follow the Adafruit tutorial. It's fantastic and almost impossible to go wrong if you follow the directions.

The Adafruit tutorials also show you how setup some Python scripts to provide simple functions on the IoT printer, including:

  • Daily weather scripts. Pretty useful at providing a local forecast.
  • Twitter script. Prints everytime someone tweets you. I immediately disabled this script.
  • Daily Sudoku puzzle. I love Sudoku and have kept this one running.

But, honestly, beyond this it's up to you to figure out interesting uses for the printer.

Not a problem for me. I had two ideas:

  1. Figure out a way to let people send messages to the printer though simple text (SMS) messages.

  2. Print out my kids daily school lunch menu. This had always been a huge annoyance as we'd have to load the school website almost every day to look it up.

Given that the thermal printer is powered by a Raspberry Pi, giving you the ability to write just about any kind of code you can think of in various languages, these ideas proved pretty simple.

Printing text messages on the IoT thermal printer

You may ask yourself, why would you want to send a text message to a printer? The best answer I can come up with is, because I can! As with most things, it's solving the challenge that provides the satisfaction, not the function itself.

As you've no doubt guessed, I used a number of cloud services to make it possible:

  • Twilio. Seriously one of the best developer services I've ever used. It's so much fun to bridge your applications with a phone (whether through SMS, MMS, or IVR).

  • Heroku. A cloud platform-as-a-service (PaaS) and part of the Salesforce1 Platform.

  • CloudAMQP. A managed RabbitMQ server in the cloud, acquired as a Heroku Add-On.

And, of course, some Python code on the Raspberry Pi.

Here's how the three cloud services work:

  1. You text a message to a phone number. This phone number is owned by Twilio and I'm effectively "renting" it. They receive the message and send it (and some additional metadata) to a web service I have running in Heroku. This is set by specifying the Messaging Request URL on your Twilio number. (Note: Please refer to Twilio's fantastic developer documentation if you have any problems.)

    Twillio Message Request URL

  2. Our web service running in Heroku receives the message and extracts the phone number and message. (Note: Heroku also has fantastic developer documentation. Please refer to it to see how to create and deploy your web service.)

  3. Using the data received from Twilio, we write a message to CloudAMQP, queuing it up for later retrieval.

  4. Before finishing, the web service on Heroku creates and sends a Twilio response so that the sender knows their message was successfully received.

    Here's the code for the web service running in Heroku:

At this point we have our message sitting in a queue. This is perfect, as it allows us to write a little code on our printer to periodically check the queue and, if a message is found, print it!

On the Raspberry Pi, there are two things I did:

  1. I created a script called amqp.py that checks for messages in the CloudAMQP queue and, if found, formats and prints the message.

  2. Modified /etc/rc.local to automatically run the amqp.py script when the Raspberry Pi is booted.

    cd /home/pi/Python-Thermal-Printer
    python amqp.py &
    

Now the printer automatically run the amqp.py script on start-up and, whenever a message is delivered, it will print it!

Let me tell you, it's super cool when you see messages start to pour in!

printerworks

Text away!

Getting the school lunch menu

As I said, I had grown tired of having to load up the school lunch menu every morning. We'd routinely have to browse to this lunch menu. It gets tiring after the 100th time.

So, with the help of my friend Steve Marx I created a Python script that does the following:

  1. Downloads all the HTML from the lunch menu website.

  2. Using a REGEX to extract the lunch menu data.

  3. Iterates through the data until it finds today's date.

  4. Formats and prints the menu on the printer.

Take a look at the code:

Simple (except for the REGEX) and effective.

Here it is in action:

2014-02-07 16 23 51

Mmm, there's nothing quite like Cowboy Mac!

There you have it! A solution guaranteed to reduce your daily stress and make your children incredibly happy!

I hope you found this solution interesting. If you do something like this yourself definitely let me know! You can find all the source code for this solution in the following gist: https://gist.github.com/wadewegner/8872919

Support for Refresh Tokens, Async Suffix, and Updated Sample Apps

Over the past two days I've committed and published a number of updates to the Salesforce Toolkits for .NET worth sharing. But first, a special thanks to Rocky Assad and Delmer Johnson for the contributions they have made to the toolkits over the past couple weeks. It's fantastic to see such passionate involvement from members of the Salesforce developer community!

There are three updates I'd like to discuss in this post:

  1. Support for refresh tokens
  2. The addition of an "Async" suffix
  3. Updated sample applications

Without futher ado ...

Support for refresh tokens

I added support for using the refresh token to acquire a fresh access token in the DeveloperForce.Common library and NuGet package. The syntax is really quite simple.

This is a simple operation. The two methods enabling this to work are TokenRefreshAsync and FormatRefreshTokenUrl.

The FormatRefreshTokenUrl method helps to create the URL with all the proper values. It is a public static method, so you can use it independently of TokenRefreshAsync. Of course, TokenRefreshAysnc uses it as well.

Notice that TokenRefreshAsync creates the same AuthToken used by the other authentication methods, so it's easy to incorporate.

Refresh Token Reminder: The refresh token may have an indefinite lifetime, persisting until explicitly revoked by the end-user. The client application can store the refresh token, using it to periodically obtain fresh access tokens, but should be careful to protect it against unauthorized access, since, like a password, it can be repeatedly used to gain access to the resource server.

The addition of an "Async" suffix

While I may not agree with the it, the Task-based Asynchronous Pattern (TAP) is very clear about the naming convention for asynchronous methods. While I believe that async should be the default (and synchronous should be the odd ball) the reality is that this is not the convention for public libraries.

Consequently, all of the methods in the toolkits have been updated to include an Async suffix.

I don't like introducing changes like this and will strive to keep them to the minimum. Fortunately, this this is simply a naming change and shouldn't be difficult for you to update.

Updated sample applications

All of our sample applications have been updated. Make sure to take a look: https://github.com/developerforce/Force.com-Toolkit-for-NET/tree/master/samples

In particular, take a look at the Windows 8 OAuth Sample to see how the token refresh is implemented.

I hope you find these updates useful!

Using the WebAuthenticationBroker for Salesforce Authentication on Windows 8

Two days ago I published a sample app for Windows Phone 8 that demonstrates how to authenticate users and make calls against the Force.com REST API using the Salesforce Toolkits for .NET. The next obvious step was to build a sample application for Windows 8. ACHIEVEMENT UNLOCKED!

Windows 8 Sample App

You can get the sample here: https://github.com/developerforce/Force.com-Toolkit-for-NET/tree/master/samples/Windows8OAuth.

To get authentication working in Windows 8 we'll use the WebAuthenticationBroker, which manages the asynchronous authentication operation in a modal dialog window. First, the user is asked to login. This is nothing more than secured browser control that renders the Salesforce login page.

Login

Once the login information has been added, the user is then asked to authorize the application.

Authorize

When this process completes, the user is returned back to the application. Programmatically we now have access to all the information returned to us from Salesforce. (Review this article if you need a refresher.)

Here's the code. When setting up your Connected App you can use the same directions I detailed in this post. Also, be sure to grab the DeveloperForce.Common NuGet package.

A few important details to point out.

  • [Line 8]: As of the writing of this post, you have to use the DisplayTypes.Popup display type instead of DisplayTypes.Touch. While the rendering isn't ideal for the modal dialog it's better than simply not working (which is what will happen if you use DisplayTypes.Touch).

  • [Line 11]: Be sure to copy your Callback Url exactly as the WebAuthenticationBroker will look for this URI and only break when it finds it.

  • [Line 28]: In addition to the access_token, refresh_token, and instance_url you can also get the id, signature, issued_at, and (if you passed it with your URL) state.

This is just the start for the Windows 8 sample. We'll add other features (such as refreshing the token and storing tokens securely) in the near future.

I hope this helps!

Connecting to Salesforce from a Windows Phone 8 App

One of the top requests I've received since releasing the Salesforce Toolkits for .NET has been a Windows Phone 8 sample application. I was very happy to oblige and today I'm pleased to share with you the Windows Phone 8 OAuth sample application. This sample makes use of the Developerforce.Force and Developerforce.Common NuGet packages. Since the .NET libraries in the toolkits are all portable class libraries developers are able to build applications targeting .NET 4 & 4.5, Windows Phone 8, Windows 8 & 8.1, and Silverlight 5.

Visual Studio Phone App

This sample application demonstrates the following:

  1. Obtaining an access token using the User-Agent flow.
  2. Manging the callback in a Windows Phone 8 app.
  3. Using the access token to make a call to the Force.com REST API.

Let's break all this down and walk through the pertinent pieces.

Creating Your Connected App

A Connected App is an application that allows you to connect to Salesforce identity and data APIs. Connected Apps use the OAuth 2.0 protocol to authenticate and acquire access tokens for use with the Force.com REST API. Additionally, the Connected App gives you granular control over the access you want to give users.

We're going to use the Connected App to give our mobile application access to the Salesforce APIs.

You can create a Connected App in your developement environment by going to Setup > Create > Apps. The last pane reads Connected Apps. Click New to create a new Connected App.

Supply the following information:

  • Connected App Name: WindowsPhoneOAuth Sample
  • Contact Email: your@emailaddress.com

Click the Enable OAuth Settings checkbox and perform the following actions:

  • Callback URL: sfdc://success
  • Selected OAuth Scopes: add them all

When complete, it should look something like this:

Connected App settings

Go ahead and save your Connected App. Salesforce will create a Consumer Key and Consumer Secret. All we need for our purposes is the Consumer Key.

Consumer Key

Note: Identity is a complex topic and it's important you know what you're doing. Just as you wouldn't store a username-password on a mobile device you shouldn't store the Consumer Secret on a mobile device. It is not as hard as you'd think to extract this information. Consequently, we will use the User-Agent flow to protect our Connected App secrets yet still let our users get a valid access token. For more details I recommend you review Obtaining an Access Token in a Native Application (User-Agent Flow).

Updating the WindowsPhoneOAuth Sample

There's only change you'll have to make in the sample application. In MainPage.xaml.cs update the ConsumerKey string to use the Consumer Key created by your Connected App. Make this change and hit F5 and give it a spin.

Login to Salesforce login
Authorize Access authorize
See Your Accounts accounts

That's it!

Of course, there's quite a bit going on behind the scenes that makes this possible. Let's take a look.

How Does this Work?

The first step is to create the URL to the Authorization Server. Fortunlately this is fairly simple as the Developerforce.Common library provides a helper method called FormatAuthUrl you can use.

Note that we use the ResponseTypes.Token response type, indicating our desire to use the User-Agent auth flow. The URL will look something like this:

https://login.salesforce.com/services/oauth2/authorize?response_type=token&client_id=3MVG9A2kN3Bn17hsEyMqRTTaEfW.t4ssmYD2zPrrftW7vokEg0kCWj3H_NwryefANj37hbxV_KyB0Qd2NLySH&redirect_uri=sfdc://success&display=touch&immediate=False&state=&scope=

The important values included her are:

  • response_type: By specifying token we indicate the User-Agent auth flow.
  • client_id: This is the Consumer Key from the Connected App.
  • redirect_uri: This is the Callback Url from the Connected App.
  • display: We've specified touch so that we get a mobile friendly login page.

Once the URL has been constructed we then tell a browser control to navigate to the URL.

Now, once we have logged in, the authorization server will redirect back with information we want to collect. We need a way to handle that callback.

Fortunately (or unfortunately, in some cases) Windows Phone 8 has what's called URI Association Schemes that auto launches apps. Since we're returning a URN with the value "sfdc://success" the URI association scheme tries to get involved and launch an application. We want to surpress this behavior while also hooking into it to get the values sent as part of the redirection.

To do this we'll do a few things:

  1. Register a protocol extension that is associated to "sfdc" (from our redirect URI).

  2. Create a AuthorizationUriMapper that will allow us to override the default MapUri behavior and get access to the information in the URN.

  3. Register our AuthorizationUriMapper

As a result of our AuthorizationUriMapper, we'll reload the MainPage.xaml and pass in the AccessToken and InstanceUrl needed to make a call the the resource server. Now making a call to the Force.com REST API to get a list of accounts is simple:

And there we have it! Access to the Force.com REST API (actually, any Salesforce API) using an access token acquired without putting any of our secrets in jeopardy.

A special thank you to several people who helped me as I built this sample: Pat Patterson for helping debug the auth flow, Matteo Pagani for helping with the ListBox binding, and Ginny Caughey & Darrel Miller for their help understanding the crazy URI assocation schemes.

I hope you find this sample useful. Good luck!

Better Support for HttpClient in the Salesforce Toolkits for .NET

If you follow me on Twitter, chances are you heard we released a set of Salesforce Toolkits for .NET last week. The reception has been extraordinary, and I'm so excited to see the applications developers build using these toolkits!

Take a look at the various annoucements and posts:

(Incidently, if you know of other posts I've missed, please let me know!)

Just because we released doesn't mean I've stopped working. In fact, I've been monitoring feedback via Twitter and Github, and today made a few worthwhile optimizations. It was rightly pointed out to me by Darrel Miller that the ServiceHttpClient did not optimally use the HttpClient library. Specifically:

  • The Developer.Common library created a new HttpClient each time. This can be an extremely expensive operation.

  • There was no way to modify the HttpRequestMessage class.

  • There was no way to use the HttpClient across ServiceHttpClient and AuthenticationClient.

To fix this, a number of changes were made to the DeveloperForce.Common library.

  1. The ServiceHttpClient now requires the HttpClient in the constructors. Given that, for most operations, the DeveloperForce.Force and DeveloperForce.Chatter libraries manage this, you will likely see no changes to any of your code. However, if you ever want to create your own HttpClient and use it to make multiple calls against the Force.com or Chatter REST APIs, you can now create it yourself and pass it in.

  2. Similarly, the AuthenticationClient class allows you to pass in the HttpClient. However, it is not required, as I'm making the assumption some people simply want to make a call to get the token, and don't care if the HttpClient is disposed.

  3. Both the ServiceHttpClient and AuthenticationClient now implement IDisposable. This means you can use Using statements to dispose the HttpClient or you can call Dispose yourself.

All these updates have been released via our NuGet packages.

Here's what it looks like in action:

I hope you find this a useful update!

Many thanks to Darrel Miller for all his great feedback!