A' y' sure? .. or experiences from a small dev shop using Azure

by Morten Hoffmann Sørensen 10. March 2016 14:32

Introduction

In our software development company, we're either working as consultants in corporate setups, as educators on courses or conferences or as project/product developers. We do most of our work using the Microsoft technology stack, and therefore we decided some time ago, that we would take the plunge, and establish development assets in the cloud using Azure. We already have experience using Amazon cloud services, so to compare the two products, the Azure adventure was needed also. Let me fill you in on our experiences so far.

Setup

Our initial and very simplistic setup was based on five subscriptions to Visual Studio Professional with MSDN (OVL licenses). We used to be a BizSpark partner but grew out of that (three year period), and had to buy into the Microsoft tool chain on ordinary market terms. Along with the MSDN subscriptions comes free Azure credits (350 DKK/month/subscription or approx. 50$), which has allowed us to experiment and gain some basic knowledge into the Azure world. More on the combined price tag later.

The overall setup looks like this.

(Besides the MSDN subscriptions we also run a Office 365 setup hosting all our email and office program needs)

Buildserver

We have a TeamCity (JetBrains) buildserver running on a small VM (Basic A1, 1.0 Core, 1.75 Gb) which is barely enough to keep us going. The estimated price tag on this baby is approx. 240 DKK/month (36$). This VM, along with our other VM’s are “classic” – Microsoft, in its wisdom, decided to differentiate VM’s created using a Service Management API (classic) from resources created by a Resource Manager (1,2). There are differences in how collected resources are treated, look to the references for further elaboration.

Another curiosity with respect to the different API’s is their visual appearance. Each has a client UI with totally different layout – why did Microsoft not just expand or evolve their first attempt?

Figure 1: "Old" Azure portal

Figure 2: "New" Azure portal

Deployment

We try to adhere to the principles of not only Continous Integration but also Continuous Delivery (CD) and have supplemented our TeamCity build server with an “Octopus” deployment server (https://octopus.com/). This server is a very tiny VM (Basic A0, 0.25 Core, 0.75 Gb) and takes care of pushing the builds from the build server to our staging environment (mostly) web sites. For instance, we have one Azure QA web site (Web App) and an external ISP hosted LIVE web site. The CD server uses an SQL Server instance for storing configuration and releases. Notice, that we’ve chosen that DEV is the local developer machine. Our first integration point then, is the build service and the QA website. With the push of a button we’re able to release a tested QA build of choice. Builds from the build server are auto promoted if all tests are green.

Repositories

We’re based on Git / Github with respect to our source code repositories. The Microsoft tools all play nicely along here and we’ve never looked back to ancient times with Subversion et al.

We would never consider going down the TFS path, so no experiences on its integration with Azure here, I’m afraid. Look to the references if you’re interested in how to leverage Azure with respect to TFS based builds (3). Then there’s the whole issue with Visual Studio Team Services, which is another Microsoft TFS initiative. But all, in all, we’ve always found the TFS setup too “corporate”.

Wiki

Besides resources needed for actual development, we have set up a DokuWiki (https://www.dokuwiki.org) for ever-changing developer related documentation. This is just an Azure Web App, and it comes for free in the “Free” tier, being a non-outbound / non-production site). We administer the site using Remote IIS Manager and FileZilla (https://filezilla-project.org/).

Bottlenecks

Price tag

So, with this relatively small company setup, how much does this cost us on a monthly basis?

MSDN (x5) is approx. 31.000 DKK/year / 12 = 2.583 DKK/month ($385/month)

Azure usage is approx. = 440 DKK/month ($65/month)

Azure credits = 350 DKK/month ($52/month)

Total(*) 2673 DKK/month (approx. $400/month)

(*) Total for five developers, excluding VAT.

This shall of course be taken with a very large grain of salt! .. our usage depends on how much development is going on, how many builds, how many deployments and so on. We’ve had 2 VM’s running around the (daytime) clock, with different loads, 1 VM (the build server) being the primary work horse. Now, you could argue that the Azure part in this equation is only minimal (some 90 DKK/month or 18 DKK/developer/month), but the only reason for us to board Azure was the MSDN prerequisites. Should we enter the Azure world on a non-credit basis the price tag starts to blur a bit. Try and figure the costs from the extremely overwhelming and confusing summery outlined for instance here (5,6).

Learning curve

The real cost of Azure however, is the learning curve associated with the mix of different tools you will have to master. We spent many hours trying to figure out infrastructure problems related to issues such as geo location of assets, different subscriptions or problems related to our VPN connection.

Penny pinching

And with respect to geo location, note that you need to make sure to establish your SQL database resources in the same geo region as the using resources, otherwise you’ll pay extra for the intermediate traffic between the database and the backend (4).

Sort of conclusion

When we first set out, we were pretty naïve on how to use our individual MSDN licenses. We thought that the total Azure credits could just be pooled and used to the benefit of our company. We soon discovered that this is indeed not the case – credits are meant exclusively for the subscription on which it originates. We asked Microsoft about this, but they don’t seem likely to change this behavior (7). This means, that we need to carefully plan on which subscription we deploy which resources, to make the most of our credits. For a big company, the Azure credits are of course just peanuts, and this whole issue deserves no mentioning, but we think, as (current) representatives of companies less than 10 developers, that Microsoft could look into how to embrace companies like us. Remember, all companies start out small.

So, all in all, yes we’re sure - we see no other future than continuing to strengthen our Microsoft (as well as AWS) cloud efforts and make the most of our experiences to the benefit of our customers.

References

  1. https://azure.microsoft.com/en-us/documentation/articles/resource-manager-deployment-model/
  2. https://www.datapipe.com/blog/2016/01/13/azure-portal-vs-azure-classic-portal-a-tale-of-two-apis/
  3. https://azure.microsoft.com/da-dk/documentation/articles/cloud-services-continuous-delivery-use-vso/
  4. http://www.hanselman.com/blog/PennyPinchingInTheCloudWhenDoAzureWebsitesMakeSense.aspx 
  5. https://azure.microsoft.com/en-us/pricing/
  6. https://azure.microsoft.com/en-gb/documentation/articles/azure-subscription-service-limits/
  7. https://feedback.azure.com/forums/344565-azure-stack/suggestions/11722554-let-small-companies-pool-their-per-user-msdn-credi

Tags:

Azure

Writing a small parser / interpreter (Part 3: Semantic processing and simple WPF MVVM UI)

by Morten Hoffmann Sørensen 19. January 2013 23:57

In the previous two steps we crafted a scanner and a parser for a subset of the LOGO programming language:

But, besides recognizing tokens and legal sentences, we also need to put semantic meaning into the LOGO programs. 

Semantic recordings

For this purpose we will use semantic records. A semantic record is introduced to remember context while parsing the LOGO sentences.

Take a look at the following (parser) code-snippet:
private ILogoCommand ParseLogoSentence()
{
  ILogoCommand result = null;
  Token nextToken = scanner.NextToken();
  switch (nextToken)
  {
    case Token.FORWARD:
    case Token.BACK:
    case Token.LEFT:
    case Token.RIGHT:
      Match(nextToken);
      Match(Token.NUMBER);
      var numberRecord = new NumberRecord(nextToken, scanner.ScanBuffer);
      if (nextToken == Token.FORWARD || nextToken == Token.BACK)
      {
        result = new LogoMoveCommand(numberRecord);
      }
      else
      {
        result = new LogoTurnCommand(numberRecord);
      }
      break;
:

When parsing simple LOGO constructs, we're using a NumberRecord to remember the number associated with a distance or angle and we're using LogoCommands to compose a record for later interpretation. One could argue that the NumberRecord seem redundant in this simple case, but actually we're using the NumberRecord for deciding the direction represented by the number.

public class NumberRecord
{
  public int Number { get; private set; }
 
  public NumberRecord(Token directionToken, string numberAsString)
  {
    Number = int.Parse(numberAsString);
    if (directionToken == Token.BACK || directionToken == Token.RIGHT)
    {
      Number = (-1)*Number;
    }
  }
}

The LogoCommands stores direction and angle information and provide means for calculating the geometry behind the sentence (its semantic meaning) through an interface:

public interface ILogoCommand
{
  TurtleSituation CalculateSituation(TurtleSituation currentSituation);
}

A TurtleSituation is just a placeholder remembering the current state of the geometry behind the LOGO program.

public class TurtleSituation
{
  private static readonly DefaultTurtleSituation defaultSituation = new DefaultTurtleSituation();
  public virtual int Angle { get; set; }
  public virtual int TurnAngle { get; set; }
  public virtual Position Position { get; set; }
 
  public static DefaultTurtleSituation DefaultSituation
  {
    get { return defaultSituation; }
  }
 
  #region Nested type: DefaultTurtleSituation endregion#
}

Should we at some point need more complex sentences supporting e.g. routines, variables or the like, we could introduce a symbol-table to store identifier information. When dealing with more complexity it is also common to push the semantic records on an (explicit) semantic stack making it possible for the parser methods to query information discovered previously. One could of course also use the C# stack, pushing records as method arguments – in fact, we do that when returning LogoCommands from the parser methods.

MVVM WPF UI

All though the premise for these blogs were parsing techniques it would be a shame not to take the last step, writing a consumer of the LogoCommand records produced by the parser.

So, with the choice of a WPF MVVM kind of UI and a bit of Cartesian geometry, the Logo parser gets a graphical face. The MVVM pattern helps separate the UI view from the view logic using merely data binding and simple commands.

Please look to the code at the end of the blog for details.

Overview and recap

Over this series we have created a layered design utilizing simple parsing techniques. The scanner transforms received characters to tokens and the parser adds meaning to the sentences outputting commands. The commands are interpreted and ends up in simple line drawing instructions shown on the UI.

The above illustration hides details like how the ViewModel exposes the actual command that calls the application logic and invokes the parser. Also, the overall data bindings between the View and the ViewModel (not just the command binding) is left out for brevity.

Check out and modify the source, and please chip in and improve the CodePlex Open Source project at http://mhslogo.codeplex.com

Tags:

Craftsmanship | Parsing

Writing a small parser / interpreter (Part 2: Parser)

by Morten Hoffmann Sørensen 7. December 2012 16:17

In the previous step, we looked at how to write a lexical scanner for a small subset of the LOGO programming language.

Our parser relies heavily on the fact that we're parsing a CFG language free from left-recursion (non-terminal not repeated as first symbol on lhs) and common left side prefixes (for any one rule, no productions share a common prefix). These properties enables us to write a special kind of parser, called a LL(1) parser.

The first "L" means we're recognizing LOGO programs read left-to-right.

The second "L" signifies that we're recognizing left-most parse-trees (parse-trees are the derivations of the production rules in our grammar), but this is non-important in our case, since the LOGO subset grammar can only produce one kind of parse-trees (you could try writing down derivations, if in doubt).

The "(1)" part tells us that it is sufficient to look one symbol ahead to be able to predict the next production rule.

You could possibly read up on the subject, preferably in Crafting a compiler with C by Fisher & LeBlanc or just take my word for it.

Tests

Unexpected token's are treated as syntax errors and therefore a suite of tests could look like this (please improve if you find them inadequate):

[TestFixture]
public class TestLogoParser
{
	[Test]
	public void CanParseSimpleLogoProgram()
	{
		TestParseProgram("FORWARD 50");
	}

	[Test]
	public void CanThrowOnSimpleSyntaxError()
	{
		var exception = TestParseProgramThrows("50 FORWARD");
		Assert.AreEqual(LogoErrorCode.SentenceError, exception.ErrorCode);
		Assert.AreEqual("50", exception.ScanBuffer);
	}

	[Test]
	public void CanThrowOnSecondSentenceWithSyntaxError()
	{
		var exception = TestParseProgramThrows("FORWARD 50 LEFT");
		Assert.AreEqual(LogoErrorCode.MatchError, exception.ErrorCode);
		Assert.AreEqual("LEFT", exception.ScanBuffer);
	}

	[Test]
	public void CanThrowOnMissingRBracketSyntaxError()
	{
		var exception = TestParseProgramThrows("REPEAT 4 [ FORWARD 50");
		Assert.AreEqual(LogoErrorCode.MatchError, exception.ErrorCode);
		Assert.AreEqual("50", exception.ScanBuffer);
	}

	private static void TestParseProgram(string program)
	{
		var parser = new LogoParser(new LogoScanner(program));
		parser.ParseLogoProgram();
	}

	private static LogoSyntaxErrorException TestParseProgramThrows(string program)
	{
		return Assert.Throws<LogoSyntaxErrorException>(
			() => TestParseProgram(program)
		);
	}
}

Code

The trick is to write a method, either a recursive one or one that "decends" into a sub-method, where each method corresponds to the possible rule outcomes for a nonterminal symbol. We match tokens as we go along.

public class LogoParser
{
	private readonly LogoScanner scanner;

	public LogoParser(LogoScanner logoScanner)
	{
		scanner = logoScanner;
	}

	// <logo-program>  ::= <logo-sentence> { <logo-sentence> } <EOF>
	public void ParseLogoProgram()
	{
		ParseLogoSentence();
		while (true)
		{
			switch(scanner.NextToken())
			{
				case Token.FORWARD:
				case Token.BACK:
				case Token.LEFT:
				case Token.RIGHT:
				case Token.REPEAT:
					ParseLogoSentence();
					break;

				default:
					Match(Token.EOF);
					return;
			}
		}
	}

	// <logo-sentence> ::= FORWARD <integer>
	//                   | BACK <integer>
	//                   | LEFT <integer>
	//                   | RIGHT <integer>
	//                   | REPEAT <integer> [ <logo-sentence> { <logo-sentence> } ]
	private void ParseLogoSentence()
	{
		var nextToken = scanner.NextToken();
		switch(nextToken)
		{
			case Token.FORWARD:
			case Token.BACK:
			case Token.LEFT:
			case Token.RIGHT:
				Match(nextToken);
				Match(Token.NUMBER);
				break;

			case Token.REPEAT:
				Match(nextToken);
				Match(Token.NUMBER);
				Match(Token.LBRACKET);
				ParseLogoSentence();
				Match(Token.RBRACKET);
				break;

			default:
				SyntaxError(String.Format("Expected one of: FORWARD, BACK, LEFT, RIGHT or REPEAT but found {0}", 
					TokenHelper.TokenToText(nextToken)), LogoErrorCode.SentenceError);
				break;
		}
	}

	private void Match(Token token)
	{
		Token nextToken = scanner.Scan();
		if (nextToken != token)
		{
			SyntaxError(String.Format("Expected {0} but found {1} (at '{2}')",
				TokenHelper.TokenToText(token),
				TokenHelper.TokenToText(nextToken), 
				scanner.ScanBuffer), LogoErrorCode.MatchError);
		}
	}

	private void SyntaxError(string errorMessage, LogoErrorCode errorCode)
	{
		throw new LogoSyntaxErrorException(errorMessage, errorCode, scanner.ScanBuffer);
	}
}

We'll also extend the LogoScanner class with the "lookahead" feature (no tests shown):

public Token NextToken()
{
	var oldIdx = idx;
	var result = Scan();
	idx = oldIdx;
	return result;
}

Ok, so far so good!

We have now created a parser and scanner capable of recognizing our LOGO subset programming language. But we need to translate the LOGO program into meaningful drawing primitives.

We will look into that in the final part 3 of this small mini-series.

Tags:

Craftsmanship | Parsing

Writing a small parser / interpreter (Part 1: Scanner)

by Morten Hoffmann Sørensen 30. November 2012 13:33

Perhaps some time in your career you've been asked to write a small "conversion tool", one which "shall perform a simple translation" of some text input and produce a text output. So, you grab the compiler or scripting tool of your choice and hack away at some regex parser thingy, right?!

There is an alternative that at first glance seems daunting, but for most cases is pretty simple and definitely worthwhile.

In the following three part sequel I will introduce the technique of writing a simple parser for a subset of the LOGO programming language in C#.

LOGO(small) syntax

LOGO is mostly known to be a toy language aimed at introducing kids and newcomers to programming. LEGO programs directs a small turtle (or arrow or whatever symbol kids find amusing) around the screen.

Here's a small sample drawing a quadrant 100 units wide.

REPEAT 4 [FORWARD 100 LEFT 90]

Expressing the legal sentences in our small subset of LOGO programs can be done using a CFG (Context-Free-Grammar) and (Backus-Naur-Form), or EBNF (Extended-BNF) where appropriate.

A CFG consists of rules transforming nonterminal symbols into a sequence of nonterminal or terminal symbols. "No context" is allowed on the left hand side (the nonterminal "definition"). Deriving legal sentences consists of replacing nonterminals with their right hand side "productions" until only terminal symbols are left.

Consider the following grammar for our LOGO subset:

<logo-program>  ::= <logo-sentence> { <logo-sentence> } EOF

<logo-sentence> ::= FORWARD <integer>

                  | BACK <integer>

                  | LEFT <integer>

                  | RIGHT <integer>

                  | REPEAT <integer> [ <logo-sentence> { <logo-sentence> } ]

<integer> are just your normal integer representation, no 0 prefixes allowed (besides 0 of course). { .. } is EBNF notation for "optional". "|" means "or". EOF means no more input characters (Aka "End-Of-File").

The above subset excels when it comes to the absence of complexities (ambiguities), "left recursion" or "similar right hand side prefixes" and is therefore suitable for "recursive decent parsing".

Boring? Don't worry, if you have made it this far, from now on no more theory.

Transforming the simple grammar above into a parser / interpreter involves three major steps:

  1. Write a scanner capable of recognizing the keywords and integers on  the input stream
  2. Write a parser that matches the recognized terminal symbols as part of the production rules
  3. Decorate the parser with semantic methods interpreting the LOGO language

Step 1: Scanner

Writing a scanner consists of recognizing and matching chars in a string – it's that simple.

An initial test (cough, a bit too elaborate to qualify for an actual TDD step, but you will get the idea) might look something along the lines of:

[Test]
public void CanRecognizeTokens()
{
	const string input = "REPEAT 4 [ FORWARD 100 LEFT 90 ] ";
	var expectedTokens = new[] { Token.REPEAT, Token.NUMBER, Token.LBRACKET, 
Token.FORWARD, Token.NUMBER, Token.LEFT, 
Token.NUMBER, Token.RBRACKET, Token.EOF };
	var scanner = new LogoScanner(input);
	foreach (var expectedToken in expectedTokens)
	{
		var token = scanner.Scan();
		Assert.AreEqual(expectedToken, token);
	}
}

And the accompanying code could look like:

public enum Token
{
	[TokenAsText("REPEAT")]
	REPEAT,
	[TokenAsText("FORWARD")]
	FORWARD,
	[TokenAsText("BACK")]
	BACK,
	[TokenAsText("LEFT")]
	LEFT,
	[TokenAsText("RIGHT")]
	RIGHT,
	[TokenAsText("NUMBER")]
	NUMBER,
	[TokenAsText("[")]
	LBRACKET,
	[TokenAsText("]")]
	RBRACKET,
	EOF
}
public class LogoScanner
{
	readonly Token[] reserved = new[] { Token.FORWARD, Token.BACK, Token.RIGHT, Token.LEFT, Token.REPEAT };

	private readonly string rawContents;
	private string scanBuffer;
	private int idx;
	private char ch;

	public LogoScanner(string input)
	{
		rawContents = input;
	}

	public Token Scan()
	{
		while (idx < rawContents.Length)
		{
			ch = rawContents[idx];
			if (ch == '[')
			{
				idx++;
				return Token.LBRACKET;
			}
			else if (ch == ']')
			{
				idx++;
				return Token.RBRACKET;
			}
			else if (char.IsDigit(ch))
			{
				scanBuffer = ch.ToString();
				idx++;
				while (idx < rawContents.Length)
				{
					ch = rawContents[idx];
					if (char.IsDigit(ch))
					{
						scanBuffer += ch;
						idx++;
					}
					else break;
				}
				return Token.NUMBER;
			}
			else if (char.IsLetter(ch))
			{
				scanBuffer = ch.ToString();
				idx++;
				while (idx < rawContents.Length)
				{
					ch = rawContents[idx];
					if (char.IsLetter(ch))
					{
						scanBuffer += ch;
						idx++;
					}
					else break;
				}
				Token lookup;
				if (LookupReserved(scanBuffer, out lookup))
				{
					return lookup;
				}
				LexicalError();
			}
			else if (char.IsWhiteSpace(ch))
			{
				idx++;
			}
			else
			{
				LexicalError();
			}
		}
		return Token.EOF;
	}

	private void LexicalError()
	{
		throw new LogoScannerException(String.Format("Lexical error at '{0}' ('{1}')", ch, scanBuffer));
	}

	private bool LookupReserved(string s, out Token lookup)
	{
		lookup = TokenHelper.LookupAsText(s);
		return reserved.Contains(lookup);
	}
}

Next, we'll look at writing the parser, utilizing the scanners token recognizing capabilities.

Tags:

Craftsmanship | Parsing

Om Strongminds

Strongminds arbejder med kundespecifik softwareudvikling til industri, web og mobil og leverer tilhørende serviceydelser, rådgivning og projekter.

Læs mere

Overblik