Connected Pawns

August 2, 2016

My First Logic App

Filed under: Azure, BizTalk — Tags: , , — mbrimble @ 6:11 pm

I really like this logic app pattern.

image

The NZ Post parcel events application (https://www.nzpost.co.nz/business/developer-centre/tracking-notification-api/watch-method) will not repeat an event. I decided to expose http endpoint in an azure Logic App and send these to a Azure Service Bus. I then I retrieve the message using a BizTalk SB-Messaging adapter, whenever I am ready and then do all the heavy lifting.

This was a brilliant experience with a customer in front of me. All done in less than 15 minutes!

July 16, 2016

BAM: “..One of more database(s) appears corrupted..”

Filed under: BizTalk — Tags: — mbrimble @ 6:06 am

I got this error.

image

Checking the event log I saw a BAM Web Service Error like

System.Web.Services.Protocols.SoapException: There are instances with duplicate ID ‘\\MyServer\customer-b2c-dw_wsl_delta201604131734004.xml’ in activity ‘GenericCountOfTrackedMessages’. The duplicates must be removed from the database to fix this problem.

Running the following query showed the duplicates in two of the partitions.

SELECT * FROM [BAMPrimaryImport].[dbo].[bam_GenericCountOfTrackedMessages_20ED6B1B_734C_4980_B213_792B16DF734B] where ActivityID IN (SELECT ActivityID FROM dbo.bam_GenericCountOfTrackedMessages_00C024D3_BCC3_4E02_A428_3358E482BF4F)

image

I decided to delete these duplicates from one of the partitions with

DELETE

–SELECT *

FROM [BAMPrimaryImport].[dbo].[bam_GenericCountOfTrackedMessages_20ED6B1B_734C_4980_B213_792B16DF734B] where ActivityID IN (SELECT ActivityID FROM dbo.bam_GenericCountOfTrackedMessages_00C024D3_BCC3_4E02_A428_3358E482BF4F)

This worked and the query of this BAM activity now works without the above error.

June 7, 2016

The parameter is incorrect (WinMgt) error when refreshing the BizTalk Group Hub

Filed under: BizTalk — mbrimble @ 9:32 am

If you want to know one way of solving this see https://social.msdn.microsoft.com/Forums/en-US/ddc32037-b449-4ed8-903a-99fc69986ede/winmgt-error-when-refreshing-group-hub?forum=biztalkgeneral

wimgt

May 21, 2016

BizTalk 2016 CTP1–BizTalk Health Monitor(BHM)

Filed under: BizTalk — Tags: — mbrimble @ 1:58 pm

It is a miserable rainy Saturday morning in Auckland and I decided to have look BizTalk 2016 CTP1. I had set up two SQL servers, many Always on instances and then configured a BizTalk server. Before going further to set up the SQL Always On availability groups I wanted to check that i had configured everything correctly. Thus I configured the BHM from the Support tools folder that comes with BizTalk 2016 CTP1 in the same way that Sandro has previously described for BizTalk 2013R2. I discovered that the BHM that comes with BizTalk 2016 CTP1 is a very old version.

old BHM2

This showed me that everything was Ok but I had not set up the back up job for the databases.

I thought it would be interesting to see if the latest version of the BHM v3.2 could be run on BizTalk 2016 CTP1. I suspected I might get an amusing critical error and I did as shown below. BHM v3.2 installed without error and the latest repository update downloaded easily. All the functionality BHM v.3.2 seemed to work but I have not tried to run terminator yet. This is pleasing because I think that the BHM is a very useful tool for support analysts to use.

image

In summary the latest version of the BHM seems to work with BizTalk 2016 CTP1. The .Net version warning made me laugh but it is understandable that the repository update does not know about BizTalk 2016 CTP1 yet. I hope that the final version of the BHM ships with v 3.2 + instead of the original version and that the repository update starts to know about BizTalk 2016.

Blast from the Past: BizTalk Orchestration Throttling Pattern

Filed under: BizTalk — Tags: , — mbrimble @ 9:23 am

Some us still do classical BizTalk Server development and recently I was faced with an old age problem namely “limiting the number of calls to a web service”. This Oracle CRM on Demand service would only allow a maximum of 5 concurrent calls. Up until now we had just limited the number of connections with an entry in the BizTalk Server configuration file. In the past we did not care if requests had to retry a couple of times because there was no in order or latency requirements. Recently a new requirement required a low latency response from this service and we could not tolerant timeouts or any retries.

I was pleased to find an old article by Seroter that show us what to do. This is based on an old BizTalk HotRod magazine article by Scott Colestock which is unfortunately no longer available. I have placed a copy of this old article in my archive here.

Time has not affected this pattern and soon we had it all working. We choose to throttle the connections that did not require low latency to 3 and were surprised to find the a sudden burst of requests processed in about the same time with and without the throttler. Clearly BizTalk still processes this very quickly but this time without retrying. This left at least two connections available for our low latency requests. This will do for now as soon we will move our endpoints to Salesforce which allows more concurrent connections.

WSL.CRMonDemand.CustomerDataBCCallerWSL.CRMonDemand.ThrottleCustomerDataB2C

May 19, 2016

Warning: BizTalk 2013 R2 is unsupported if you install .Net 4.6

Filed under: BizTalk — mbrimble @ 6:24 pm

A new critical warning has appeared in the BizTalk Health Monitor for all of our BizTalk 2013 R2 servers after the latest repository update from Microsoft as shown in the screenshot below.

image

How did this occur? We built our BizTalk servers using .Net 4.5. This is the supported version of .Net for BizTalk servers. On investigation we discovered windows patching had installed a recommended security update (Microsoft .NET Framework 4.6.1 is available on Windows Update and WSUS) that installs .Net 4.6. The windows patching does not occur automatically and the server team applied this patch because they thought it was recommended by Microsoft.

clip_image002[6]

There is good reason as to why they have installed 4.6 as Microsoft support has ended for 4.5.1. “Support Ending for the .NET Framework 4, 4.5 and 4.5.1. As previously announced, starting January 12, 2016 Microsoft will no longer provide security updates, technical support or hotfixes for .NET 4, 4.5, and 4.5.1 frameworks. All other framework versions, including 3.5, 4.5.2, 4.6 and 4.6.1, will be supported for the duration of their established lifecycle. The decision to end support for these versions will allow us to invest more resources towards improvements of the .NET Framework.”Microsoft .NET Framework 4.6.1 is available on Windows Update and WSUS.

What was missed is that .Net version 4.5.2 is supported going forward.

We have been running without any obvious errors since 18 March , so what it he big deal.

We considered 3 mitigations to this critical warning;

1) Ask Microsoft to certify that .Net 4.6 runtime is supported.

2) Attempt to uninstall and block  KB3102467.

3) Do nothing and carry on.

I initially thought we should ignore this warning and carry on and not do anything because the Microsoft warning is not correct. I reasoned that the .Net version 4.6 is supported at runtime and that .Net version 4.5 is only supported at compile time. We compile all our code with the .Net version code set to 4.5 because this is a Microsoft requirement. The .net 4.6 runtime is backward compatible with .net 4.5 and .net 4.5 assemblies should run without any issues. This is what we have observed because we have been running with the.Net 4.6 version for many months now without any known issues.  A sequitur on this is that the 4.6 runtime is safe as long as the compile time is set to 4.5. Unfortunately the official word is that “BizTalk Server 2013 R2 only supports .Net framework 4.5.2 . As of now if we are on 4.6X  then you are on unsupported version.”

Thus our only option is to uninstall .Net 4.6 and reinstall .Net 4.5.2 if we want to be supported. How risky is that?

I wonder how many other people would admit to falling into this trap. Please reply to this email me or reply to this blog  if you have. Go check the .Net versions that are running on your BizTalk 2013 R2 servers now.

POSTSCRIPT added 31/5

We have successfully removed .Net 4.6 and blocked future upgrades from .Net 4.5.2. by the uninstalling the KB3102467.

May 1, 2016

BizTalk Server 2016 CTP: BizTalk 2013 BAM Alerts configuration error is now fixed.

Filed under: BizTalk — mbrimble @ 11:45 am

I was pleased to find that you no longer get an error like “Cannot alter the role ‘NSSubscriberAdmin’, because it does not exist or you do not have permission.” when you try to configure BAM Alerts when you configure BizTalk 2016. This error first appeared in BizTalk 2013 and was not fixed in BizTalk 2013R2. See http://blog.brauwers.nl/2013/03/23/biztalk-2013-rtm-installation-and-configuration-issues-i-encountered-and-how-to-fix-them/ for the BizTalk 2013 error. This error is now fixed in BizTalk 2016.

image

March 29, 2016

Azure Service Bus Relays, SAS tokens and BizTalk Server

Filed under: Azure, BizTalk — Tags: , — mbrimble @ 11:52 am

Many people have written about Azure Service Bus Relays in the past and a summary can be found here. Dan Rosanova recently tweeted “….We’re trying to discourage ACS for security. SAS is our preferred model.”. The ACS security pattern is described here and the SAS pattern is described here. This article attempts to summarise BizTalk adapter support for using SAS tokens.

Most BizTalk Server examples use ACS tokens rather than SAS tokens, probably because the BizTalk Adapters only allowed configuration with ACS tokens when service bus relays were first released with BizTalk 2013. BizTalk 2013 R2 has limited support for configuration of SAS tokens and most adapters only allow use of ACS tokens out of the box (OOTB). If you want to use a SAS token you have to be very inventive. I hope that BizTalk vNext will add SAS token support for all WCF adapters.

The BizTalk 2013 R2 SB-messaging adapter allows configuration of SAS tokens. The diagrams below shows the BizTalk 2013 SB-messaging adapter that only allows configuration of ACS tokens and the BizTalk 2013 R2 adapter that uses SAS tokens.

image

The BizTalk WCF-BasicHttp and WCF-NetTcp adapters can be configured to use a relay binding in two ways but only ACS tokens are supported out of the box. The two configurations are ;

  1. Selecting the in built WCF-BasicHttpRelay or WCF-NetTcpRelay adapters. See https://msdn.microsoft.com/en-us/library/jj572837.aspx and https://msdn.microsoft.com/en-us/library/jj572861.aspx. In BizTalk 2013R2 during configuration you can only choose a ACS token.image
  2. Using WCF Service Publishing Wizard choosing the  WCF-WSHttp, WCF-WebHttp, WCF-BasicHttp or WCF-CustomIsolated adapter and selecting  the Add a Service Bus checkbox and you will be presented with additional screens at the final stages of the wizard that allows you to choose a service bus namespace and specify the ACS credentials that BizTalk uses to connect to the relay.

image

I want to finish this article by highlighting that with a little bit of effort you can indeed use a service bus relay for BizTalk 2013 R2 secured with SAS tokens with other WCF adapters. We have used WCF-WebHTTP adapters with service bus relays secured with SAS tokens, following a pattern first created by Johann Cooper ( https://adventuresinsidethemessagebox.wordpress.com/2015/04/12/getting-azure-service-bus-relays-and-the-biztalk-wcf-webhttp-adapter-to-play-nice-with-each-other/). Johann said “I’ve found a way to get this to work with SAS, but more on this in another blog post”. I am going to tell you how we did it in case I forget.

Firstly we downloaded this version of the Microsoft.ServiceBus.dll and installed this in the GAC

image

Secondly we added a behaviour extension to the 32 and 64 bit machine.config files.

        <add name="transportClientEndpointBehavior" type="Microsoft.ServiceBus.Configuration.TransportClientEndpointBehaviorElement, Microsoft.ServiceBus, Version=2.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

Thirdly we added a webHttpRelayBinding as a binding extension to the same machine config files.

       <add name="webHttpRelayBinding" type="Microsoft.ServiceBus.Configuration.WebHttpRelayBindingCollectionElement, Microsoft.ServiceBus, Version=2.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

Finally we edited the web configuration of the WCF-WebHTTP adapter create following the article above and added a sharedAccessSignature as tokenProvider in the endpointBehaviour (circled in yellow). Once you browse to the base URL a service bus relay protected with SAS Tokens is created in Azure. We created a BizTalkAccessKey that has manage, listen and send permissions and a ClientAccessKey that has only listen permission.

image

In this article I have shown that while you can use SAS tokens with the BizTalk Server 2013 R2 WCF adapters with some customisation. I look forward to a BizTalk Server release that allows us to configure SAS tokens on the WCF Adapters.

March 8, 2016

BizTalk Host Settings – “Default application domain for isolated adapter”

Filed under: BizTalk — Tags: , , , — mbrimble @ 9:18 pm

What does this setting do? In this post I describe why I found this setting useful and ask what are the disadvantages are of turning this setting to on.

image

 

I have a BizTalk Server 2013 R2 that uses eight WCF-WebHTTP request response adapters with service bus relays that are hosted in the same isolated host. All of these receive locations use the same WCF Custom behaviours, BRE pipeline component, JSON encode pipeline component, JSON decode pipeline component that are configured in different ways. Some of the ports have maps configured and others have no maps configured. We have observed a memory leak on this server which causes some of the in–process and isolated hosts to throttle after a long time. The typical warning we see in the logs is  “BizTalk host ,<hostname> throttled because ProcessMemory exceeded the configured throttling limit.”. These warning only go away when we brutally restart IIS. Stopping and starting the application pools or any of the in-process hosts does not fix the issue. We collected performance counters and then ran the PAL analyzer.  What we found was a steady increase in memory used by the .NET CLR but it did seem to be specific to any host instance.

MEM1

I reviewed all the code that i thought could be running on the receive locations and could not find any reason for a memory leak. I was at the point of running at running DebugDiag to get a dump to try to find the source of the memory leak when one of my colleagues, Leela Pachaiyappan spotted an error in the event logs that said

There was a failure executing the receive pipeline: “Microsoft.BizTalk.DefaultPipelines.PassThruReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35” Source: “Unknown ” Receive Port: “CustomerAPI” URI: “/Service.svc” Reason: Attempted to access an unloaded AppDomain.”

I found this thread https://msdn.microsoft.com/en-us/library/aa577833.aspx that tells us how fix this permanently. The solution to this issue is “ by enabling Default application domain for isolated adapter in the Isolated hosts settings….The SOAP adapter runs in the IIS process space. If more than one Web service exists in the IIS AppPool, then every Web service ends up having its own AppDomain. By default all messaging engine objects are created in the first AppDomain (that is,. the AppDomain corresponding to the first Web service). If the first Web service is inactive for an extended period of time for any reason, IIS unloads the first AppDomain. When this happens, all services in the hosting process become unusable.”. I was really interested in this because I reasoned I was probably loading the same assemblies in many different AppDomains. If they were all loaded into the same AppDomain then the common assemblies would be shared and would this led to a decrease my memory usage. Furthermore would the spin up from idle be quicker because the assemblies for the services are always pinned in memory?

I had never heard of this setting before and started to research it. Other blogs have repeated this advice but no one spelled out what the disadvantages of switching this setting is. I asked around amongst my colleagues and the best they could come up with was “Normally having separate app domains provides isolation. In the .net world this provided a way to manage security between different layers in your application.“. I reasoned that this was not of concern to me and I decided to set the Default application domain for isolated adapter to true.

Once again we collected the performance counters and monitored the server over the next week. We did not observe any throttling because of exceeding ProcessMemory and the request response latency was marginally better. What was more surprizing was that we no longer observed a memory leak as shown below!

MEM2

 

In summary I have found that enabling Default application domain for isolated adapter in the Isolated hosts settings has mitigated a memory leak on our BizTalk Server. The only disadvantage that I can see is that some assemblies are now permanently pinned in memory and occupy some of the memory on your server. What I cannot understand is why this setting is not set to true by default for every BizTalk Server Isolated host.

I wish to acknowledge Mike Howell for collecting and analysing all the performance counters.

February 20, 2016

Processing Zero Byte Files with BizTalk Server

Filed under: BizTalk — mbrimble @ 9:45 pm

This post attempts to summarise what happens when some BizTalk Adapters picks up a zero byte file. I am using BizTalk Server 2013 R2 with CU2 for these tests

First let’s make it clear what zero byte file I used for my testing. I created my test file by opening notepad and saving as an XML file without adding anything to it. If you open the file in a binary editor it looks like this

image

Second let’s set up a receive location that uses a file adapter as shown below.

image

If we drop zero-byte file above into the pick up folder the following warning is recorded in the event log.

Log Name:      Application
Source:        BizTalk Server
Date:          17/02/2016 9:39:43 p.m.
Event ID:      7182
Task Category: BizTalk Server
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:     
Description:
The FILE receive adapter deleted the empty file "D:\brimble1\Test\zero.xml" without performing any processing.

The File receive adapter deletes zero byte files by design. Many people have wanted to process zero byte files and have created a custom file adapter for this task e.g. http://ninocrudele.me/2012/07/05/biztalk-and-zero-byte-file/ . Why would you want to process a zero byte file? In most cases this is an annoyance because a zero byte would fail in any pipeline that contains a XML Disassembler. In many cases I have had applications that randomly create zero byte files and the design of the BizTalk Server has been a god send because I do not have to implement any special pipeline component to handle this situation.

What does the BizTalk Server FTP adapter do if it picks up a zero byte file? Let’s set up a receive location with a FTP adapter and pick up the same zero byte file as shown below;

image

This time the file suspends in BizTalk Server the group hub because I had no subscription to this receive port. This is OK because it shows that the zero byte file has got into the message box and if we wanted we could do something with it. This is annoying if you are not using a pass through pipeline because invariably the zero byte file will error in the pipeline.

ftp

Now what happens if we subscribe to this receive port with a send port that uses file adapter and  a pass through pipeline? The zero byte file gets written to the destination folder.

How do other adapters behave with zero byte files?  CU1 for BizTalk Server 2013R2 contains a fix for SFTP adapter cannot send zero-byte file in BizTalk Server 2013 R2 or BizTalk Server 2013  but I have not tried the SFTP receive adapter with zero bytes files. This author has struggled with a HTTP receive adapter that would happily deliver zero byte files.

In summary while the receive file adapter deletes empty files the receive FTP adapter does not. If you want to pick up zero-byte files then I think you can use the FTP adapter instead of the FILE adapter. If you want to do this this then you will have to use a custom pipeline or an orchestration for further processing.

Older Posts »

Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 214 other followers