I was trying to deploy multiple resources using ARM Template deployment and got interrupted with the following error.

New-AzResourceGroupDeployment : 11:59:25 PM – Error: Code=InvalidTemplateDeployment; Message=The template deployment ‘AzureFunctionsVPN’ is not valid according to the validation procedure. The tracking id is ’16b65e77-8c46-462f-8f4c-a909492f7229′. See inner errors for details.

It is very common to see such errors while working with ARM Template deployments. I found the following command-let extremely helpful without which it would have been a herculean task.

Get-AzLog -CorrelationId 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' -DetailedOutput

If you are still using old RM command-lets, use the following

Get-AzureRMLog -CorrelationId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -DetailedOutput

The above command generates a lengthy JSON Output, pay attention to Properties > Status Message > Message

Note: Please give at least a minute before you run the above command, it usually takes that much time for the log to show up, otherwise you might get an empty response sometimes.

You can also get the logs based on a date and time range. For more information please refer to the following link.

https://docs.microsoft.com/en-us/powershell/module/az.monitor/get-azlog?view=azps-3.8.0

I received the following error while installing Azure PowerShell Modules.

Install-Module -Name Az -AllowClobber

PackageManagement\Install-PackageProvider : No match was found
for the specified search criteria for the provider 'NuGet'.
The package provider requires 'PackageManagement' and 'Provider'
tags. Please check if the specified package has the tags.

Issue:

The issue, as I understand it, is that PowerShell by default uses TLS 1.0 for web requests and causing an issue in this case. So when the Install-Module command is trying to install the missing Package-Provider as a pre-requisite and AZ Modules itself, it’s failing! See step 3 to change the default to TLS 1.2

Solution:

Step 1: Check the PowerShell version installed and update it if required.

$PSVersionTable.PSVersion

Note: Azure PowerShell works with PowerShell 5.1 or higher on Windows, or PowerShell Core 6.x and later on all platforms. You should install the latest version of PowerShell Core available for your operating system. Azure PowerShell has no additional requirements when running on PowerShell Core.

To use Azure PowerShell in PowerShell 5.1 on Windows:

1. Update to Windows PowerShell 5.1 if needed. If you’re on Windows 10, you already have PowerShell 5.1 installed.
2. Install .NET Framework 4.7.2 or later.
3. Make sure you have the latest version of PowerShellGet. Run Update-Module PowerShellGet -Force.

Step 2: Check if the NuGet package provider is installed

Get-PackageProvider -ListAvailable

Following is what I have initially, notice NuGet package provider is missing.

Step 3: Run the following command

The following will for the PowerShell to use TLS 1.2 

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Step 4: Re-Run Install-Module command

Install-Module -Name Az -AllowClobber

Note: The above command should have installed the missing package-provider as well and should look like the following. See highlighted.

References:

https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-3.8.0

This post acts as a quick reference for all the popular features that differentiate Service Bus from Storage Queues.

This blog post assumes that you have some fundamental understanding of messaging options within Azure. But if you want to get started and need a more detailed understanding please refer to the following articles.

Microsoft Service Bus

Messaging options in the Microsoft Azure Platform

Service Bus feature summary:

Note: All the above options applies to both Queues and Topics.

Time to live:
Message time to live determines how long a message will stay in the queue before it expires and is removed or dead lettered. When sending messages it is possible to specify a different time to live for only that message. This default will be used for all messages in the queue which do not specify a time to live for themselves. 

Lock Duration:
Sets the amount of time that a message is locked for other receivers. After its lock expires, a message pulled by one receiver becomes available to be pulled by other receivers. Defaults to 30 seconds, with a maximum of 5 minutes.

Duplicate Detection:
Enabling duplicate detection configures your queue to keep a history of all messages sent to the queue for a configurable amount of time. During that interval, your queue will not accept any duplicate messages. Enabling this property guarantees exactly-once delivery over a user-defined span of time

Dead Lettering:
Dead lettering messages involves holding messages that cannot be successfully delivered to any receiver to a separate queue after they have expired. Messages do not expire in the dead letter queue, and it supports peek-lock delivery and all transactional operations.

Sessions:
Service bus sessions allow ordered handling of unbounded sequences of related messages. With sessions enabled a queue can guarantee first-in-first-out delivery of messages.

Partitioning:
Partitions a queue across multiple message brokers and message stores. Disconnects the overall throughput of a partitioned entity from any single message broker or messaging store. This property is not modifiable after a queue has been created.

References:

Time to live:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-expiration
Duplicate Detection:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection
Dead-Letter-Queues:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues
Sessions:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-sessions
Partitioning:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-partitioning

I have been meaning to get around to this article for a long time, this blog post is for developers or folks who wants to read through the clutter, get a ten thousand foot view of what Azure Storage is, when to use what service and various API options available. I assume that you have some understanding what Azure is and familiar with some jargon with Azure Storage as well.

What is Azure storage? (Understanding with analogy)

A perfect analogy would be, let’s assume that you are relocating from Texas to Virginia, you said your good-byes, wrapped up your boring office send-offs, took pictures, packed all your stuff and  bagged a few memories. You drove to Virginia and negotiating with a Storage Service Provider like U-Haul, or EzStorage to store all your stuff. You might have boxes of all sizes – small, big and some extra large; fragile items, wall hangings, suite cases, may be an old piano or an antic car etc. Everybody will have unique needs based on what they have to store. And these storage providers will allocate space based on what you want to store and how you want to store. You might need a garage for your car or your might just need a small shelf shared with others for storing small boxes. Of course! you pay for their service, you pay based on the size, kind of storage, location of the storage etc. Azure Storage Services is no different.

Now relate Azure Storage with the Storage Facility, Azure Storage Services also stores stuff for you, unlike boxes or wall hangings, you might have VHDs (Virtual Hard Disks), Big & Small images, High definition videos, Office documents etc. whatever ever your needs are, Azure will accommodate them for you. Azure provides various Storage Services to host different files types and catering to various use cases. 

Let us get started!  The following section will provide use cases for understanding on when to use different storage options.

1. Table Services

2. Blob Services

3. Queue Services

4. File Share Services

Table Service
Typical use cases:

• For basic NoSQL/Key-Value functionality and the ability to scale quickly.
  • NoSQL (Not only SQL) is revolutionary and the underlying concept of Big Data. 
  • What’s Big Data? In layman’s terms, when the concept of storage started, humans used to contribute data, but with an evolution of millions of smart gadgets like smart TVs, smart refrigerators, smart cameras everything is interconnected and all of these products contribute to Terabytes if not Petabytes of data.
    If your applications allocate resources processing real-time data,  you might lose some of the data being fed to these systems continuously, you need a way to quickly “somehow” save this data in some format and process it later. This ‘somehow’ is NoSQL which means that you need a type of storage that can quickly take what every is inputted and stored for later processing.
    When the demand for resources is reaching its limit (while adding real-time data), underlying systems automatically scale out and scale up if needed. This is where Azure will help you.
• If you have a need to input and quickly retrieve millions of entities (similar to rows in SQL) this is a good choice.
  • If you are not following relational schema it becomes a whole lot easier to input and extract data.
    (Note: You should have Row and Partition key to fetch data fast or else as the entities grow querying becomes slow.)
• If your data storage requirements exceed 150 GB and you are reluctant to manually shard or partition your data.
• If you want to keep your costs low. Table storage services are comparatively cheaper than other storage services .
  • Note: Microsoft has been optimizing Azure SQL Database service to reduce overall costs and also ongoing administration costs, with a change like this, more and more customers are preferring SQL Database to Table Service.
• If you require robust disaster recovery capabilities that span geographical locations.

Avoid:

• If your data has complex relationships, in other words, data that require RDBMS (go with SQL Azure).
• You if you need the support of secondary indexes, this is not an option.
• If your code depends on ADO.NET or ODBC.

Blob Service (Block blob & Page Blob)
Typical use cases

• If you have tons of static files like pictures, videos, pdf, office documents etc. this is the default choice.
  • As a part of modernization, more companies are migrating to the cloud, it could be websites, applications or even whole infrastructure. There might be hundreds, thousands or even millions of supporting files. Azure Blob service can host these files for you.
  • Depending on the size of the file you are trying to store, you have two options
    • Block Blob
      • Each Block Blob can be up 200 GB, you can upload block blobs with a size up to 64 MB in one operation.
    • Page Blob.
      • Each Page blob can be up to 1 TB in size and consist of a collection of 512-byte pages. You set the maximum size when creating a page blob and then you can write or update specific pages
• This is most popular among all Storage Services, as a matter of fact Azure uses this for all of its storage needs, this could be while you create a VM and storing any supporting files/data used by other services on Azure.
• When people refer to migrating Virtual Machines to Cloud, generally they prep the machine by running tools like Sysprep on those machines and uploading VHDs to BLOB service.

Queue Service(Azure Queues & Service Bus queues)
Typical use cases

• If you desire to have a loosely coupled or applications that can interact asynchronously.
  • For instance, you have a system that checks inventory and notifies what needs to be ordered and you have another system that does the ordering for you.What if the order processing VMs or Worker Roles are offline? What happens if one of them is busy or overloaded with traffic? How do you recover from failure? Queue Services will help you with these scenarios
• If you have a need for large numbers of messages that needs to be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS.
• Load Leveling & Balancing
  • If you have apps that experience a huge burst of traffic, you could level the load if you could categorize processes that can wait (in the queue) until a free worker becomes available.
  • For load balancing,  you could spin up additional worker roles to process messages and balance the load among the worker processes. Also, this can be automated.
• Subscription
  • If you have one producer and multiple consumers, Queues are an excellent choice. For instances, one system can generate a messages (Producer) and multiple systems that subscribe and consume the messages (Consumers) and processes them independently.
• Scheduled Processing
  • If you do not want to stress your applications during the peak traffic hours, you can isolate that task that can be processed later on a scheduled basis. This could be nightly or weekly, you can add messages to the Queue and process them based on the schedule.

Note: Azure Queues and Service Bus Queues work differently, it is good to understand the differences between these two and make informed architectural decisions. Refer to ‘References’ section for good articles.

Advice on Multiple Queues:
I am not against using multiple queues, but use them sparingly, the idea is, you can achieve a lot with single queues itself (if architected well).
As a matter of fact, I have used multiple queues myself in the following use cases
1. Different Priorities: You want different priorities for different messages. The last thing you want is a high-priority message getting stuck behind hundreds of low-priority messages. Create a hi-priority queue for these.
2. Large Transactions: If the number of message transactions (Read,Put,Delete) exceed over 500 transactions per second. Creating multiple queues will spread the transaction volume across storage partitions.

File Storage
Typical use cases

• Share data across on-premises and cloud servers
  • If you have a hybrid set up and you have applications that share data across on-premise and cloud.
• Migrate file share-based applications
  • You can migrate applications to the cloud with no code changes.
• Integrate modern applications
  • With more devices accessing the same information and distributed globally, it creates a dire need for sharing resources and build more file sharing applications. Also, it supports REST API for building modern applications.
• High Availability
  • If all you need is a file share interacting via Server Message Block 3.0 (SMB) protocol it is highly efficient to use Azure File Share service backed by Azure infrastructure, redundancy, and SLA.
• Sharing files between Services
  • If you have VMs or different services that need to share files this is an excellent choice.

Note: If your application is heavy on using file attributes, file processing and not just regular read/write, there are certain unsupported features.
Click Here for the complete list.

After all these services are provisioned, there should be an option to consume these services. Following screen capture provides a quick view or a cheat sheet of what Storage API provides. Most of the operations are self-explanatory based on the naming convention. I will strive to write more articles on each one of these and provide examples in C# leveraging REST API.

Azure Storage Services Quick Reference:

Conclusion:

I was confused on when to use what service, but after working hands on and a lot of reading, I was able to assimilate some information but not everything. I made an honest attempt to provide a quick read and to get an overall view of what Azure Storage Services are, what to choose when and when to avoid. Also concluded with summarizing most of the operations that you can do with via Azure Storage API. Hope this article made some sense and helped you with something.

References:

[1] Azure Queues and Service Bus queues – compared and contrasted

[2] Azure Storage Queues VS Azure Service Bus Queues

Explaining Azure in simple terms can be a minute conversation, but if you attempt to deep dive into various services, it can even take months. No kidding, I am serious!

Microsoft provides over 200 Services, and you would be spell-bound on the rate at which new features get added on a daily basis.

This blog post does not get into the weeds of each of the services,  but will cover few basic fundamentals that will help you understand the terminology and assimilate some of the technical jargon.

Before even starting on what Azure is, it is a good idea to talk a little about Cloud, as Azure itself is a Microsoft’s flavor of Cloud. There are many other Cloud service providers such as Amazon Web Services, Akamai, CenturyLink, CSC, Dimension Data, Fujitsu, Google, IBM (SoftLayer),  Interoute,  Joyent, Rackspace, Verizon, Virtustream, VMware etc. and the list goes on…

So what is Cloud? Why companies want to be on Cloud? Why there are companies providing these services?

Technology plays a very key role in managing business operations and customer relations.

Following image demonstrates Azure replacing some or all of the On-Premise services.

Azure replacing On-Premise Services

Shift towards Cloud is not always about monetary reasons, it is also about ease of use and peace of mind.  This is one of the million reasons why companies are leaning more towards Cloud Computing and designing their own Cloud strategy.

Cloud has been a ubiquitously ambiguous. There is no true definition of what qualifies a company as a Cloud Service provider. For instance, one service provider may provide only half of the service provided by its competitors.

For instance, Rackspace does a very good job on providing IaaS services and they could provide solid references from their clientele for those services. Overall as a customer you would assume that these Service Providers would at least provide IaaS / PaaS services.

On a lighter note, you cannot run couple of servers off of your basement and call yourself a Cloud provider. In reality, you should be able to scale-up or scale-out dynamically catering to the demand, security, ability to adhere to SLA of 99.9%  up-time and much more. It is not an easy task. Cloud is a serious business!

Enough said, lets get started on some of the terminology used with in Azure and may be common to most of the Cloud Service providers.

1. Public, Private and Hybrid Cloud

Public Private and Hybrid Cloud Offerings

2. IaaS, PaaS, SaaS

Following is just a thirty thousand foot view, explaining in detail would require a complete blog post by itself.

If you are interested in knowing more in depth please follow this article.

As a matter of fact, with the Cloud maturing day by day the differences are getting blurrier.

IaaS vs. PaaS vs. SaaS

3. Azure Services

Everything in Azure is termed as a ‘Service’. It could be a VM or Website or CDN (Content Distribution Network).

Azure services are categorized as show below,  to further explore please follow this article.

Azure Services

4. Azure Portals

Microsoft provides easy to use web portals to leverage their services. Usually there are two portals, one in production and one is New portal. The intention of the new portal is to keep up with the ever changing user experience demands and making it mobile friendly. They are both meant for the same purpose. Following are some of the key differences

• Not all features are migrated into the new portal, some of them may be deprecated or merged with other features.
• New Portal has RBAC (Role Based Access Management)
  • The current production portal does not have this feature, which means that if you are logged-in, you are an Admin. Scary!
  • With RBAC you can associate roles to the users to have read only, write , contribute or even delegate admin role.
• More modern user interface, even more mobile friendly.

1. Production URL: https://manage.windowsazure.com/
   
   

   Quick Peek : Window Azure Production Portal

2. Preview Portal: https://portal.azure.com/
   
   

   Quick Peek: Azure Preview Portal

5. Subscriptions

Azure Subscriptions

6. Pricing Calculator

This is a very useful tool to price and configure Azure services for your scenarios. If you have a Pre-Paid subscription your credit status will appear as an overlay on top of window.

Credit Status

Following is the screen capture of pricing calculator, indicating how much it would cost to add ‘DocumentDB’ as a service.

Pricing Calculator

I hope you enjoyed reading this article and hope this was useful!!