Design, Manage and Monitor – Microsoft Azure Storage Services

I have been meaning to get around to this article for a long time, this blog post is for developers or folks who wants to read through the clutter, get a ten thousand foot view of what Azure Storage is, when to use what service and various API options available. I assume that you have some understanding what Azure is and familiar with some jargon with Azure Storage as well.

What is Azure storage? (Understanding with analogy)

A perfect analogy would be, let’s assume that you are relocating from Texas to Virginia, you said your good-byes, wrapped up your boring office send-offs, took pictures, packed all your stuff and  bagged a few memories. You drove to Virginia and negotiating with a Storage Service Provider like U-Haul, or EzStorage to store all your stuff. You might have boxes of all sizes – small, big and some extra large; fragile items, wall hangings, suite cases, may be an old piano or an antic car etc. Everybody will have unique needs based on what they have to store. And these storage providers will allocate space based on what you want to store and how you want to store. You might need a garage for your car or your might just need a small shelf shared with others for storing small boxes. Of course! you pay for their service, you pay based on the size, kind of storage, location of the storage etc. Azure Storage Services is no different.

Now relate Azure Storage with the Storage Facility, Azure Storage Services also stores stuff for you, unlike boxes or wall hangings, you might have VHDs (Virtual Hard Disks), Big & Small images, High definition videos, Office documents etc. whatever ever your needs are, Azure will accommodate them for you. Azure provides various Storage Services to host different files types and catering to various use cases. 

Let us get started!  The following section will provide use cases for understanding on when to use different storage options.

1. Table Services

2. Blob Services

3. Queue Services

4. File Share Services

Table Service
Typical use cases:

• For basic NoSQL/Key-Value functionality and the ability to scale quickly.
  • NoSQL (Not only SQL) is revolutionary and the underlying concept of Big Data. 
  • What’s Big Data? In layman’s terms, when the concept of storage started, humans used to contribute data, but with an evolution of millions of smart gadgets like smart TVs, smart refrigerators, smart cameras everything is interconnected and all of these products contribute to Terabytes if not Petabytes of data.
    If your applications allocate resources processing real-time data,  you might lose some of the data being fed to these systems continuously, you need a way to quickly “somehow” save this data in some format and process it later. This ‘somehow’ is NoSQL which means that you need a type of storage that can quickly take what every is inputted and stored for later processing.
    When the demand for resources is reaching its limit (while adding real-time data), underlying systems automatically scale out and scale up if needed. This is where Azure will help you.
• If you have a need to input and quickly retrieve millions of entities (similar to rows in SQL) this is a good choice.
  • If you are not following relational schema it becomes a whole lot easier to input and extract data.
    (Note: You should have Row and Partition key to fetch data fast or else as the entities grow querying becomes slow.)
• If your data storage requirements exceed 150 GB and you are reluctant to manually shard or partition your data.
• If you want to keep your costs low. Table storage services are comparatively cheaper than other storage services .
  • Note: Microsoft has been optimizing Azure SQL Database service to reduce overall costs and also ongoing administration costs, with a change like this, more and more customers are preferring SQL Database to Table Service.
• If you require robust disaster recovery capabilities that span geographical locations.

Avoid:

• If your data has complex relationships, in other words, data that require RDBMS (go with SQL Azure).
• You if you need the support of secondary indexes, this is not an option.
• If your code depends on ADO.NET or ODBC.

Blob Service (Block blob & Page Blob)
Typical use cases

• If you have tons of static files like pictures, videos, pdf, office documents etc. this is the default choice.
  • As a part of modernization, more companies are migrating to the cloud, it could be websites, applications or even whole infrastructure. There might be hundreds, thousands or even millions of supporting files. Azure Blob service can host these files for you.
  • Depending on the size of the file you are trying to store, you have two options
    • Block Blob
      • Each Block Blob can be up 200 GB, you can upload block blobs with a size up to 64 MB in one operation.
    • Page Blob.
      • Each Page blob can be up to 1 TB in size and consist of a collection of 512-byte pages. You set the maximum size when creating a page blob and then you can write or update specific pages
• This is most popular among all Storage Services, as a matter of fact Azure uses this for all of its storage needs, this could be while you create a VM and storing any supporting files/data used by other services on Azure.
• When people refer to migrating Virtual Machines to Cloud, generally they prep the machine by running tools like Sysprep on those machines and uploading VHDs to BLOB service.

Queue Service(Azure Queues & Service Bus queues)
Typical use cases

• If you desire to have a loosely coupled or applications that can interact asynchronously.
  • For instance, you have a system that checks inventory and notifies what needs to be ordered and you have another system that does the ordering for you.What if the order processing VMs or Worker Roles are offline? What happens if one of them is busy or overloaded with traffic? How do you recover from failure? Queue Services will help you with these scenarios
• If you have a need for large numbers of messages that needs to be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS.
• Load Leveling & Balancing
  • If you have apps that experience a huge burst of traffic, you could level the load if you could categorize processes that can wait (in the queue) until a free worker becomes available.
  • For load balancing,  you could spin up additional worker roles to process messages and balance the load among the worker processes. Also, this can be automated.
• Subscription
  • If you have one producer and multiple consumers, Queues are an excellent choice. For instances, one system can generate a messages (Producer) and multiple systems that subscribe and consume the messages (Consumers) and processes them independently.
• Scheduled Processing
  • If you do not want to stress your applications during the peak traffic hours, you can isolate that task that can be processed later on a scheduled basis. This could be nightly or weekly, you can add messages to the Queue and process them based on the schedule.

Note: Azure Queues and Service Bus Queues work differently, it is good to understand the differences between these two and make informed architectural decisions. Refer to ‘References’ section for good articles.

Advice on Multiple Queues:
I am not against using multiple queues, but use them sparingly, the idea is, you can achieve a lot with single queues itself (if architected well).
As a matter of fact, I have used multiple queues myself in the following use cases
1. Different Priorities: You want different priorities for different messages. The last thing you want is a high-priority message getting stuck behind hundreds of low-priority messages. Create a hi-priority queue for these.
2. Large Transactions: If the number of message transactions (Read,Put,Delete) exceed over 500 transactions per second. Creating multiple queues will spread the transaction volume across storage partitions.

File Storage
Typical use cases

• Share data across on-premises and cloud servers
  • If you have a hybrid set up and you have applications that share data across on-premise and cloud.
• Migrate file share-based applications
  • You can migrate applications to the cloud with no code changes.
• Integrate modern applications
  • With more devices accessing the same information and distributed globally, it creates a dire need for sharing resources and build more file sharing applications. Also, it supports REST API for building modern applications.
• High Availability
  • If all you need is a file share interacting via Server Message Block 3.0 (SMB) protocol it is highly efficient to use Azure File Share service backed by Azure infrastructure, redundancy, and SLA.
• Sharing files between Services
  • If you have VMs or different services that need to share files this is an excellent choice.

Note: If your application is heavy on using file attributes, file processing and not just regular read/write, there are certain unsupported features.
Click Here for the complete list.

After all these services are provisioned, there should be an option to consume these services. Following screen capture provides a quick view or a cheat sheet of what Storage API provides. Most of the operations are self-explanatory based on the naming convention. I will strive to write more articles on each one of these and provide examples in C# leveraging REST API.

Azure Storage Services Quick Reference:

Conclusion:

I was confused on when to use what service, but after working hands on and a lot of reading, I was able to assimilate some information but not everything. I made an honest attempt to provide a quick read and to get an overall view of what Azure Storage Services are, what to choose when and when to avoid. Also concluded with summarizing most of the operations that you can do with via Azure Storage API. Hope this article made some sense and helped you with something.

References:

[1] Azure Queues and Service Bus queues – compared and contrasted

[2] Azure Storage Queues VS Azure Service Bus Queues