Saturday, January 10, 2015
Thursday, January 1, 2015
It is that time of year where I take a step back and reflect on the previous year and then start thinking about the upcoming year. 2014 was an incredible year, one that I will not forget any time soon. I did a lot of travelling in 2014 and met a lot of great people, for this I am grateful for the opportunities that allowed this to happen.
So let’s get into it, here are some of the events that took place over the past year.
Released another book
Johan Hedberg, Morten la Cour and myself released a second edition of the (MCTS) Microsoft BizTalk Server (70-595) certification book to include new BizTalk 2013 content. We released the book in March, 2014 as it also co-incided with the launch of the Microsoft Partner Assessment exam that Microsoft uses to measure its Silver or Gold Application Integration partners.
If your are interested in getting certified, either personally or as part of the Microsoft Partner network, I still feel that it is a great resource that will aid you in achieving your goal.
In March, I had the opportunity to head to Europe to present at a few events. These events included the BizTalk Summit in London on March 3rd/4th, Swedish BizTalk User group in Stockholm, Sweden on March 5th and the Netherlands BizTalk User group on March 6th.
Special thanks to Saravana Kumar, Johan Hedberg and Steef-Jan Wiggers for allowing me to participate in these different events in your countries. The BizTalk community is second to none in Europe so it is always an honour to head over there.
Even though it was an extremely busy week, we did have some time to see the sights in London. But otherwise a lot of time was spent at airports and in airplanes.
After my return from Europe, I made a career change and joined MuleSoft. Event though MuleSoft has its headquarters in San Francisco I was able to sign on as a Remote Solutions Architect.
In 2014, MuleSoft made a lot of investments in supporting the Microsoft eco-system. Most of the activities I was involved with included Microsoft at some level. The highlight of my time there would have to be getting involved in the release of a couple of investments related to calling .NET code (.Net Connector) from a Mule workflow and the MSMQ connector. I had the opportunity to demonstrate these components at MuleSoft’s flagship event called Connect in San Francisco in May 2014.
I decided to leave MuleSoft at the end of September 2014. The logistics of me being a remote worker in Canada created some challenges and as a result I decided to move on. Overall, I value the time I spent at MuleSoft. I got to learn about a new eco-system, met a lot of great, smart people and got exposure to API Management which I hadn’t been exposed to previously.
After leaving Mulesoft, I traded my eclipse IDE for Visual Studio. I have shifted my attention back to the Microsoft Integration stack including BizTalk, Azure Service Bus and Azure API Management.
I was able to attend the Annual MVP Summit in Redmond. It was another great opportunity to talk shop with my integration colleagues from around the globe. Since the Summit has moved to the fall instead of Spring, we have developed a bit of a tradition by heading to Lowell’s, in the Pike Market, for breakfast before heading to a Seattle Seahawks game. Even though it rained pretty much the whole game it was still a fun way to to spend a Sunday ffternoon.
Microsoft had a bit of a surprise for us once the sessions started. It was at this point that we started to hear more about the next generation platform that they were building. This more than kept us occupied in many discussions through-out the week.
The amount of brain-power that shows up for MVP Summit is flat-out impressive. Some of the brightest minds in the industry are present and eager to share some of their experiences. Mark Mortimore, who runs marketing for BizTalk, decided to tap into some of this knowledge and have people record some short demos in advance of the Integrate 2014 event. I took this opportunity to discuss a scenario I had built in the spring involving capturing data from a Power Plant and making it available to mobile devices via BizTalk, Azure Service Bus and Azure Mobile Services. You can view my clip and others here.
Microsoft held its annual Integration event in North America in December 2014. I had the opportunity to speak at this event. My topic involved considerations when choosing an Integration Platform. The Integration Platform landscape has been very crowded lately and there are a lot of mixed messages flying around. The point of my session was to take a step back an look at a requirements based process to selecting a platform. I discussed some of the top requirements that I would be looking for in a platform.
I also took this opportunity to announce a Whitepaper that myself, Michael Stephenson and Steef -Jan Wiggers had been working on called Choosing the right Integration Platform. This whitepaper is available for free on the BizTalk360 site.
The Integrate 2014 event had an incredible amount of announcements and gave attendees a lot of insight into where Microsoft is headed with Integration. You can find a more detailed recap of the event here.
During all of these activities, I was also pursuing a Master’s Degree. In December 2014, I graduated from Arizona State University with a 4.0 GPA. The program that I was enrolled in was called MSIM (Masters of Science in Information Management). The program is delivered on-campus and online through the W.P. Carey School of Business. For the past 16 months I have spent approximately 20 hours a week in the program.
It was a great experience and I am glad that I had the opportunity to go through it. I had some great teammates in my group and we worked really well together even though both of them are in Phoenix and I am in Calgary.
In 2015, I am definitely looking for a slower pace and being able to spend more time with my family. While I will continue to focus on technology with my spare cycles, I definitely plan to spend more time skiing and running this year. Between school and doing a lot of travel last year these are some of the activities that I had to concede.
From a technology perspective, I certainly plan to dive deeper into the upcoming Microsoft Microservices offering, Azure API Management, Azure Event Hubs and Stream Analytics.
I did receive word that my MVP status has been renewed for 2015. I want to thank both my Canadian MVP leads Sim and Joel and the Product Group leadership team Vivek, Guru and Mark for keeping me in the program even though I spent some time away in another platform. 2015 is going to be a really big year for Microsoft in the Integration space and I am thrilled to get involved.
I also want to take this opportunity to thank all my twitter followers and readers of this blog for their continued interest in what I have to say. The Microsoft Integration community is really amazing and I am inspired to be part of it. Here’s to a great 2015!
Sunday, December 7, 2014
I have recently made it home from a great week at Redmond’s Microsoft campus where I attended the Integrate 2014 event. I want to take this opportunity to thank both Microsoft and BizTalk360 for being the lead sponsors and organizers of the event.
I also want to call out to the other sponsors as these events typically do not take place without this type of support. I think it also a testament of just how deep Microsoft’s partner ecosystem really is and it was a pleasure to interact with you over the course of the week.
Speaking at the event
I want to thank Microsoft and BizTalk360 for inviting me to speak at this event. This was the first time that I have had the chance to present at Microsoft’s campus and it was an experience I don’t think I will ever forget. I have been to Microsoft campus probably around 20 times for various events but have never had the opportunity to present. It was a pretty easy decision.
One of the best parts of being involved in the Microsoft MVP program is the international network that you develop. Many of us have been in the program for several years and really value each other’s experience and expertise. Whenever we get together, we often compare notes and talk about the industry. We had a great conversation about the competitive landscape. We also discussed the way that products are being sold with a lot of buzzwords and marketecture. People were starting to get caught up in this instead of focusing on some of the fundamental requirements. Much like any project should be based upon a formal, methodical, requirements driven approach, so should buying an integration platform.
These concepts introduced the idea of developing a whitepaper where we would identify requirements “if I was buying” an integration platform. Joining me on this journey was Michael Stephenson and Steef-Jan Wiggers. We focused on both functional and nonfunctional requirements. We also took this opportunity to rank the Microsoft platform, which includes BizTalk Server, BizTalk Services, Azure Service Bus and Azure API Management. Our ranking was based upon experiences with these tools and how our generic integration requirements could be met by the Microsoft stack. This whitepaper is available on the BizTalk360 site for free. Whether your are a partner, system integrator, integration consultant or customer you are welcome to use and alter as you see fit. If you feel we have missed some requirements, you are encouraged to reach out to us. We are already planning a 1.1 version of this document to address some of the recent announcements from the Integrate event.
My presentation focused on 10 of the different requirements that were introduced in the paper. I also included a ‘Legacy Modernization’ demo that highlights Microsoft’s ability to deliver on some of the requirements that were discussed in the whitepaper. This session was recorded and will be published on the BizTalk360 site in the near future.
Disclaimer: What I am about to discuss is all based upon public knowledge that was communicated during the event. I have been careful to ensure what is described is accurate to the best of my knowledge. It was a fast and furious 3 days with information moving at warp speed. I have also included some of my own opinions which may or may not be inline with Microsoft’s way of thinking. For some additional perspectives, I encourage you to check out the following blog posts from the past week:
- Integrate 2014 – Final Thoughts by Nick Hauenstein
- Azure BizTalk Microservices, initial thoughts by Sam Vanhoutte
- Microsoft making big bets on Microservices by Saravana Kumar
- BizTalk Server, Microservices and APIs by Gijs in ‘t Veld
There was a lot of build up to this event, with Integration MVPs seeing some early demos there was cause for a lot of excitement. This spilled over to twitter where @CrazyBizTalk posted this prior to the event kicking off. The poster(I know who you are ) was correct, there has never been so much activity on twitter related to Microsoft Integration. Feel free to check out the timeline for yourself here.
The ever so popular Scott Guthrie or otherwise known as “Scott Gu” kicked off the Integrate 2014 event. Scott is the EVP of Microsoft’s Cloud and Enterprise groups. He provided a broad update on the Azure platform describing all of the recent investments that have been rolled out.
Picture Source @SamVanhoutte
Some of the more impressive points that Scott made about Azure include:
- Azure Active Directory supports identity federation with 2342 SaaS platforms
- Microsoft Azure is the only cloud provider in all 4 Gartner magic quadrants
- Microsoft Azure provides the largest VMs in the cloud known as ‘G’ Machines (for Godzilla). These VMs support 32 cores, 448 GB of Ram and 6500 GB of SSD Storage
- Microsoft is adding 10 000+ customers per week to Microsoft Azure
For some attendees, I sensed some confusion about why there would be so much emphasis on Microsoft Azure. In hindsight, it makes a lot of sense. Scott was really setting the stage for what would be come a conference that focused on a cohesive Azure platform where BizTalk becomes one of the center pieces.
Picture Source @gintveld
A Microservices platform is born
Next up was Bill Staples. Bill is the General Manager for the Azure Application Platform or what is also known as “Azure App Platform”. Azure App Platform is the foundational ‘fabric’ that currently enables a lot of Azure innovation and will fuel the next generation integration tools for Microsoft.
A foundational component of Azure App Platform is App Containers. These containers support many underlying Azure technologies that enable:
- > 400k Apps Hosted
- 300k Unique Customers
- 120% Yearly Subscription Growth
- 2 Billion Transactions daily
Going forward we can expect BizTalk ‘capabilities’ to run inside these containers. As you can see, I don’t think we will have any performance constraints.
Picture Source @tomcanter
Later in the session, it was disclosed that Azure App Platform will enable new BizTalk capabilities that will be available in the form of Microservices. Microservices will enable the ability provide service composition in a really granular way. We will have the ability to ‘chain’ these Microservices together inside of a browser(at design time), while enjoying the benefits of deploying to an enterprise platform that will provide message durability, tracking, management and analytics.
I welcome this change. The existing BizTalk platform is very reliable, robust, understood, and supported. The challenge is that the BizTalk core, or engine, is over 10 years old and the integration landscape has evolved with BizTalk struggling to maintain pace.
BizTalk capabilities exposed as Microservices puts Microsoft in the forefront of integration platforms leapfrogging many innovative competitors. It allows Microsoft’s customers to enable transformational scenarios for their business. Some of the Microservices that we can expect to be part of the platform include:
- Workflow (BPM)
- SaaS Connectivity
- Rules (Engine)
- Mapping (Transforms)
- API Management
Picture Source @jeanpaulsmit
We can also see where Microsoft is positioning BizTalk Microservices within this broader platform:
Picture Source @wearsy
What is exciting about this is new platform is the role that BizTalk now plays in the broader platform. For a while now, people have felt that BizTalk is that system that sits in the corner that people do not like to talk about. Now, BizTalk is a key component within the App Platform that will enable many integration scenarios including new lightweight scenarios that has been challenging for BizTalk Server to support in the past.
Whenever there is a new platform introduced like this, there is always the tendency to chase ‘shiny objects’ while ignoring some of the traditional capabilities of the existing platform that allowed you to gain the market share that you achieved. Microsoft seems to have a good handle on this and has outlined the Fundamentals that they are using to build this new platform. This was very encouraging to see.
Picture Source @wearsy
At this point the room was buzzing. Some people nodding their heads with delight(including myself), others struggling with the term Microservice, others concerned about existing requirements that they have and how they fit into the new world. I will now break down some more details into the types of Microservices that we can expect to see in this new platform.
One of the current gaps in Microsoft Azure BizTalk Services (MABS) is workflow. In the following image we will see the workflow composer which is hosted inside a web browser. Within this workflow we have the ability to expose it as a Microservice, but we also have the ability to pull in other Microservices such as a SaaS connector or a Rules Service.
Picture Source @saravanamv
On the right hand corner of this screen we can see some of these Microservices that we can pull in. The picture is a little “grainy” but some of the items include:
- Retrieve Employee Details (custom Microservice I suppose)
- Custom Filter
- Acme (custom Microservice I suppose)
- Survey Monkey SaaS Connector)
- Email (SaaS Connector)
Picture Source (@mikaelsand)
In the demo we were able to see a Workflow being triggered and the tracking information was made available in real time. There are also an ability to schedule a workflow, run it manually or trigger it from another process.
Early in the BizTalk days there as an attempt to involve Business Analysts in the development of Workflows (aka Orchestrations). This model never really worked well as Visual Studio was just too developer focused, and Orchestration Designer for Business Analysts (ODBA) just didn’t have the required functionality for it to be a really good tool. Microsoft is once again attempting to bring the Business Analyst into the solution by providing a simple to use tool which is hosted in a Web browser. I always am a bit skeptical when companies try to enable these types of BA scenarios but I think that was primarily driven from workflows being defined in an IDE instead of a web browser.
Picture Source @wearsy
Once again, nice to see Microsoft focusing on key tenets that will drive their investment. Also glad to see some of the traditional integration requirements being addressed including:
- Persist State
- Message Assurance
- End to end tracking
All too often some of these ‘new age’ platforms provide lightweight capabilities but neglect the features that integration developers need to support their business requirements. I don’t think this is the case with BizTalk going forward.
Picture Source @wearsy
A gap that has existed in the BizTalk Server platform is SaaS connectivity. While BizTalk does provide a WebHttp Adapter that can both expose and consume RESTful services, I don’t think it is enough (as I discussed in my talk). I do feel that providing a great SaaS connector makes developers more productive and reduces the time to deliver projects is mandatory. Delivering value quicker is one of the reasons why people buy Integration Platforms and subsequently having a library that contains full featured, stable connectors for SaaS platforms is increasingly becoming important. I relate the concept of BizTalk SaaS connectors to Azure Active Directory Federations. That platform boasts more than 2000+ ‘identity adapters”. Why should it be any different for integration?
The following image is a bit busy, but some of the Connector Microservices we can expect include:
- Traditional Enterprise LOBs
- Dynamics CRM Online
- SAP SuccessFactors
- Dynamics AX
- Azure Mobile Services
- Office 365
- SQL Azure
- Azure Blobs and Table
This list is just the beginning. Check out the Marketplace section in this blog for more announcements.
Picture Source @wearsy
Rules (Engines) are a component that shouldn’t be overlooked when evaluating Integration Platforms. I have been at many organizations where ‘the middleware should not contain any business rules’. While in principle, I do agree with this approach. However, it is not always that easy. What do you do in situations where you are integrating COTS products that don’t allow you to customize? Or there may be situations where you can customize, but do not want to as you may lose your customizations in a future upgrade. Enter a Rules platform.
The BizTalk Server Rules Engine is a stable and good Rules Engine. It does have some extensibility and can be called from outside BizTalk using .NET. At times it has been criticized as being a bit heavy and difficult to maintained. I really like where Microsoft is heading with its Microservice implementation that will expose “Rules as a Service” (RaaS? - ok I will stop with that). This allows integration interfaces to leverage this Microservice but also allows other applications such as a Web or Mobile applications to leverage. I think there will be endless opportunities for the broader Azure ecosystem to leverage this capability without introducing a lot of infrastructure.
Picture Source @wearsy
Once again, Microsoft is enabling non-developers to participate in this platform. I think a Rules engine is a place where Business Analysts should participate. I have seen this work on a recent project with Data Quality Services (DQS) and don’t see why this can’t transfer to the Rules Microservice.
Picture Source @wearsy
Another capability that will be exposed as a Microservice is Data Transformation (or mapping). This is another capability that will exist in a Web browser. If you look closely on the following image you will discover that we will continue to have what appears to be a functoid (or equivalent).
Only time will tell if a Web Browser will provide the power to build complex Maps. One thing that BizTalk Server is good at is dealing with large and complex maps. The BizTalk mapping tools also provide a lot of extensibility through managed code and XSLT. We will have to keep an eye on this as it further develops.
Within BizTalk Server we have Business Activity Monitoring (BAM). It is a very powerful tool but has been accused of being too heavy at times. One of the benefits of leveraging the power of Azure is that we will be able to plug into all of those other investments being made in this area.
While there was not a lot of specifics related to Analytics I think it is a pretty safe bet that Microsoft will be able to leverage their Power BI suite which is making giant waves in the industry.
One interesting demo they did show us was using Azure to consume SalesForce data and display it into familiar Microsoft BI tools.
I see a convergence between cloud based integration, Internet of Things (IoT), Big Data and Predictive analytics. Microsoft has some tremendous opportunities in this space as they have very competent offerings in each of these areas. If Microsoft can find a way to ‘stitch’ them all together they were will be some amazing solutions developed.
Picture Source @wearsy
Below is a Power BI screen that displays SalesForce Opportunities by Lead Source.
Picture Source @wearsy
Marketplace - Microservice Gallery
Buckle your seatbelts for this one!
Azure already has a market place appropriately called Azure Marketplace. In this Marketplace you can leverage 3rd party offerings including:
- Data services
- Machine Learning
- Virtual Machines
- Web applications
- Azure Active Directory applications
- Application services
You can also expect a Microservice Gallery to be added to this list. This will allow 3rd parties to develop Microservices and add them to the Marketplace. These Microservices can be monetized in order to develop a healthy eco-system. At the beginning of this blog post you saw a list of Microsoft partners who are active in the existing Integration eco-system. Going forward you can expect these partners + other Azure partners and independent developers building Microservices and publishing them to to this Marketplace.
In the past there has been some criticism about BizTalk being too .Net specific and not supporting other languages. Well guess what? Microservices can be built using other languages that are already supported in Azure including:
This means that if you wanted to build a Microservice that talks to SaaS application ‘XYZ” that you could build it in one of this languages and then publish it to the Azure Marketplace. This is groundbreaking.
The image below describes how a developer would go ahead and publish their Microservice to the gallery through a wizard based experience.
Picture Source @wearsy
Another aspect of the gallery is the introduction of templates. Templates are another artifact that 3rd parties can publish and contribute. Knowing the very large Microsoft ISV community with a lot of domain expertise this has the potential to be very big.
Some of the examples that were discussed include:
- Dropbox – Office365
- SurveyMonkey – SalesForce
- Twitter – SalesForce
With a vast amount of Connector Microservices, the opportunities are endless. I know a lot of the ISVs in the audience were very excited to hear this news and were discussing what templates they are going to build first.
Picture Source @nickhauenstein
What about BizTalk Server?
Without question, a lot of attendees are still focused on On-Premises integration. This in part due to some of the conservative domains that these people support. Some people were concerned about their existing investments in BizTalk Server. Microsoft confirmed (again) their commitment to these customers. You will not be left behind! On the flipside, I don’t think you can expect a lot of innovation in the traditional On-Premises product but you will be supported and new versions will be released including BizTalk Server 2015.
You can also expect every BizTalk Server capability to be made available as a Microservice in Azure. Microsoft has also committed to providing a great artifact migration experience that allows customers to transition into this new style of architecture.
Picture Source @wearsy
If there is one thing that I would like you to take away from this post it is the “power of the Azure platform”. This is not the BizTalk team working in isolation to develop the next generation platform. This is the BizTalk team working in concert with the larger Azure App Platform team. It isn’t only the BizTalk team participating but other teams like the API Management team, Mobile Services team, Data team and many more I am sure.
In my opinion, the BizTalk team being part of this broader team and working side by side with them, reporting up the same organization chart is what will make this possible and wildly successful.
Another encouraging theme that I witnessed was the need for a lighter weight platform without compromising Enterprise requirements. When you look at some of the other platforms that allow you to build interfaces in a web browser, this is what they are often criticized for. With Microsoft having such a rich history in Integration, they understand these use cases as well as anyone in the industry.
Overall, I am extremely encouraged with what I saw. I love the vision and the strategy. Execution will become the next big challenge. Since there is a very large Azure App Platform team providing a lot of the foundational platform, I do think the BizTalk team has the bandwidth, talent and vision to bring the Integration specific Microservices to this amazing Azure Platform.
In terms of next steps, we can expect a public preview of Microservices (including BizTalk) in Q1 of 2015. Notice how I didn’t say a BizTalk Microservices public preview? This is not just about BizTalk, this about a new Microservice platform that includes BizTalk. As soon as more information is publicly available, you can expect to see updates on this blog.