Showing posts with label Algorithms. Show all posts
Showing posts with label Algorithms. Show all posts

Tuesday, 15 March 2011

Let’s synchronize our watches: Determining Clock Skew with Christian’s Algorithm and the Reactive Framework

In every playground race, there’s always one child who sets off somewhere between “On your marks” and “Get set”, gaining an unfair advantage over the honest ones who wait for “Go!”. I faced a similar problem in the design of my Windows Phone 7 multi-player game, Simon Squared, though it was unsynchronized clocks and network latency that were to blame rather than cheats.

The multi-player mode in the game relies on puzzles being shown to all players at the same time, and players race to complete them. Since some puzzles can be solved in a matter of seconds, players who get to see them even slightly sooner than their competitors stand a good chance of getting to the solution first. So I needed a plan for avoiding the digital equivalent of playground squabbles.

Any such plan has to take into account network latency: an unpredictable amount of time can elapse between the server shouting “Go!”, and the phones hearing the message. So I got my server to send out a message announcing that it would shout “Go!” at time T, where T is 4 seconds on from the time on the server’s clock (and broadcast in UTC obviously). This gives even the most far-flung phone plenty of time to receive the message, check its own clock, and then start a count-down.

Bet you’ve spotted the flaw in that straight-away! What if the server’s clock and the phones’ clocks are out of sync?

An Overview of Christian’s Algorithm

That’s where Christian’s algorithm comes in. It gives us a simple method of broadcasting the time at the server to a client whilst allowing for network latency. It works like this:

  1. Client sends a message to the server: “What’s the time?” [adding ‘Mr. Wolf’ is optional]. Crucially, it notes the time that it sent the message (call it Tsent)
  2. Server responds as quick as it can, giving the time according to its own clock, Tserver.
  3. When Client gets the message, it notes the time of receipt (call it Treceived). Then it does some maths: the round-trip time, RTT, is Treceived – Tsent . So assuming that the server responded instantly, and that the network latency was the same in both directions, that means that the server actually sent the message RTT/2 seconds ago. Thus, at the instant Client receives the message, the time at the server is Tserver + RTT/2. Then the Client can compare with its own clock and determine the difference – the clock skew.

Since network latency can vary with each request we can get an improved result by trying the whole exercise several times then cherry picking the result which had the shortest round trip time, since that minimises the error in our estimate of the network latency.

So what does that look like in code?

The Server Side – with WCF REST

Well first we need the server to be able to serve up the time:

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceContract]
public class TimeService
{
    [WebGet(UriTemplate = "/?timeAtClient={clientReportedTime}")]
    public TimeCheck GetTime(string clientReportedTime)
    {
        return new TimeCheck()
                   {
                       ClientReportedTime = DateTimeOffset.Parse(clientReportedTime, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal),
                       TimeAtServer = DateTimeOffset.UtcNow,
                   };
    }
}

This is making use of WCF 4’s REST capabilities. Notice the WebGet attribute on the method, with it’s UriTemplate. If I add the appropriate route  to my Global.ascx file,

routes.Add(new ServiceRoute("Time", new WebServiceHostFactory(), typeof(FeedbackService)));

that will ensure that HTTP GET requests like http://localhost/MyApp/Time/?timeAtClient=2011-03-17T20:19:30.2196972Z get routed to the CheckTime method with the appropriate part of the query string passed into the clientReportedTime parameter.

So that's the server side done.

RestSharp and Rx for Asynchronous Web Requests

Then the client needs to be able to call the server. I’ve been making use of the excellent RestSharp library here:

_restClient = new RestClient(ServerUrl);

...

private IObservable<TimeCheck> GetServerTime()
{
    var request = new RestRequest("Time?timeAtClient={timeNow}");
    request.AddUrlSegment("timeNow", DateTime.UtcNow.ToString("O"));

    return ExecuteRequest<TimeCheck>(request);
}

private IObservable<TResult> ExecuteRequest<TResult>(RestRequest restRequest) where TResult : new()
{
    var subject = new AsyncSubject<TResult>();

    _restClient.ExecuteAsync<TResult>(restRequest, response => HandleResponse(response, subject));

    return subject;
}

private void HandleResponse<T>(RestResponse<T> response, AsyncSubject<T> subject)
{
    subject.OnNext(response.Data);
    subject.OnCompleted();
}

This is where things start hotting up. The first part of GetServerTime is obvious enough: in lines 7-8 I’m constructing a RestRequest consisting of a url and the appropriate query string parameter. Then I Execute it. Now since we’re talking Windows Phone here, I can’t make synchronous web requests – the phone UI has to remain responsive. So I can’t wait and then return the response from the method.

Instead I return an IObservable, an interface you might not be familiar with. This creature is the brain-child of the clever folks on the Reactive Framework team. They like to call IObservable the mathematical dual of IEnumerable. And what they mean is this: IEnumerable is used when you want to pull elements one by one out of a collection. IObservable is used when you want elements to be pushed one by one to you. You can think of an IObservable as being a stream of events. And just like you can build queries on top of IEnumerable, using Where and Select, so you can build queries on top of IObservable.

Here I’m using an IObservable as the channel to push back the result of the Web request when it becomes available – you can see that happening in the HandleResponse method, which is the callback that RestRequest calls when it gets a response back from the web server.

Finally – the Algorithm

Finally we get to the implementation of the algorithm itself: and with the underpinnings in place, it turns out to be one big LINQ query:

public IObservable<TimeSpan> DetermineClockSkew()
{
    var timeOffset = (from tick in Observable.Interval(TimeSpan.FromMilliseconds(100)).Take(5)
                      from timeCheck in GetServerTime().Timestamp()
                      let estimatedOneWayLatency =
                          (timeCheck.Timestamp.UtcDateTime - timeCheck.Value.ClientReportedTime).
                              TotalMilliseconds/2
                      let estimatedTimeAtServer =
                          timeCheck.Value.TimeAtServer + TimeSpan.FromMilliseconds(estimatedOneWayLatency)
                      let predictedClockSkew = estimatedTimeAtServer - timeCheck.Timestamp
                      select new {predictedClockSkew, estimatedOneWayLatency})
        .MinBy(p => p.estimatedOneWayLatency)
        .Select(p => p.predictedClockSkew);

    return timeOffset;
}

Let's step through this line by line:

  • Line 1: We start things off by generating a stream of 5 tick events, at 100 millisecond intervals – we’re going to ping the server 5 times, and use the result with the shortest round trip time
  • Line 2: Each time we hear a tick we fire off a what’s-the-time request to the server using GetServerTime which we discussed above. The Timestamp() method will stamp each response that we get back from the server with the time at which we received it.
  • Line 3: For each response we get back from the server we estimate the one-way latency from server to client by taking half of the round-trip time (when we send a request to the server we send our own time, and the server echoes this back in the ClientReportedTime property of the message)
  • Line 4: Now that we have an estimate of how long the message took to get from the server, we can estimate the time at the server at this precise moment.
  • Line 5: We can now figure out the difference between the clock on the phone and the server’s clock by comparing the time at which we received the server’s response with our estimate of the server’s time at that moment.
  • Line 6: We stash our estimate of the latency, and the estimate of the clock skew into an anonymous type
  • Line 7 - 8: We look over all 5 attempts at estimating the clock skew, and we take the one which had the smallest latency. Notice how MinBy returns an IObservable rather than an actual value as it might in good old Linq-to-Objects: it has to be this way, otherwise the whole method would block waiting for all the web requests to complete, rather defeating the our use of asynchronous calls elsewher.

So there you have it: thanks to Christian, the WCF REST framework and the Rx team, a neat and elegant way of determining clock skew between server and phone.

See it in Action

Remember, for a limited time only you can see this in action by downloading my game to your phone or Emulator (for free!) and playing with your friends. Full source code to the whole game is also available.

Finally, judging for the Windows Phone 7 app contest finishes on March 31st, so make sure you check out the entries, and be sure to tell Red Gate what you think of mine!

Monday, 14 June 2010

Project Euler 59: Cracking an XOR code

Code breaking is about as glamorous as it gets for a Geek. So when it comes to Project Euler Problem 59, think 24, with Jack Bauer unscathed amidst a hail of bullets, memory stick in one hand, terrorist’s neck throttled in the other, threatening terrible retribution to the bad guy’s family if the secret goes to the grave with him. Terrorist rasps out,

“There’s a file … cipher1.txt … ASCII format … urrrh … XOR … 3 … letter … password … lower case … all English”

What would Chloe O’Brian make of that when she received the file over a secure socket from Jack’s Smart Phone? Being the bright girl she is, I’ve no doubt that she’d fire up MonoDevelop (only the dumb FBI would use Visual Studio and Windoze) and write the following:

var encryptedBytes = File.ReadAllText(dataPath)
                        .Split(',')
                        .Select(byte.Parse)
                        .ToArray();

That would get her the encrypted data from the file as a sequence of bytes. Now to generate all possible passwords:

const byte FirstChar = (byte)'a';
const byte LastChar = (byte)'z';

var allPossiblePasswords = from a in FirstChar.To(LastChar)
                           from b in FirstChar.To(LastChar)
                           from c in FirstChar.To(LastChar)
                           select new[] { a, b, c };

XOR encryption requires that the password be repeated cyclically, abcabcabc.... So given a sequence containing the characters of password, how about a Repeat extension method that repeats the sequence ad infinitum? (if you look at the following code and hear the Infinite Loop alarms go off, remember that iterators are lazily evaluated):

public static IEnumerable<T> Repeat<T>(this IEnumerable<T> sequence)
{
    while (true)
    {
        foreach (var item in sequence)
        {
            yield return item;
        }
    }
}

While she’s at it, she’ll need to Xor two sequences together:

public static IEnumerable<byte> Xor(this IEnumerable<byte> sequenceA, IEnumerable<byte> sequenceB)
{
    return sequenceA.Zip(sequenceB, (left, right) => (byte)(left ^ right));
}

This uses the Zip method, new in .Net 4.0, which merges two sequences together, allowing elements from left and right sequences to be processed in pairs (if you’re surprised, as I was, that the (byte) cast is necessary, see this question on Stack Overflow).

Now she can try out all the passwords, seeing what they give her if she uses them to decrypt the text. Note that Chloe has already adopted .Net 4.0 and the PLINQ extensions, so she’s using AsParallel() to max out her 128 Core workstation. (ConvertAsciiBytesToString is another extension method that simply delegates to ASCIIEncoding.GetString).

var possibleDecryptions = from trialPassword in allPossiblePasswords.AsParallel()
                          let repeatedPassword = trialPassword.Repeat()
                          select encryptedBytes
                                          .Xor(repeatedPassword)
                                          .ConvertAsciiBytesToString();

But she’s hardly going to spend the next 24 hours trawling through the possible decryptions to pick out the one in recognisable English from the 17575 in gobledygook. So she needs some means of assessing them automatically. This is where Chloe is forever in my debt. She read last weeks episode of Functional Fun, so she knows about my LanguageAnalyser that uses frequency analysis to determine how likely a sample of text is to be written in a particular language.

Rather than using the analyser to determine which of several languages the sample is written in, Chloe can use her prior knowledge that the coded message is written in English, and determine which of the possible decryptions has letter frequencies that most closely match those in English.

var languageAnalyser = new LanguageAnalyser();

var decryptionsWithDistanceToEnglish = from decryptedString in possibleDecryptions
                                       select new
                                                  {
                                                      DecryptedString = decryptedString,
                                                      Distance = languageAnalyser.DetermineCloseness(decryptedString, LanguageProfiles.English)
                                                  };

var mostLikelyDecryption = decryptionsWithDistanceToEnglish.MinItem(p => p.Distance);

And that's it. Thanks to Linguistic Geometry and LINQ, Chloe can report the decrypted (and surprisingly biblical!) message to Jack, so that he can get on with saving the world.

Ahem. Before we got carried away we were trying to solve Project Euler Problem 59, and that asks for the sum of the Ascii values of the decrypted message. So we need one more step:

var sumOfAsciiValues = mostLikelyDecryption.DecryptedString
                        .ConvertStringToAsciiBytes()
                        .Aggregate(0, (runningTotal, next) => runningTotal += next);

The full code is shown below, and is also available on MSDN Code Gallery.

namespace ProjectEuler
{
    [EulerProblem(59, Title="Using a brute force attack, can you decrypt the cipher using XOR encryption?")]
    public class Problem59
    {
        public long Solve()
        {
            const byte FirstChar = 97;
            const byte LastChar = 122;

            var dataPath = @"ProblemData\Problem59.txt";

            var encryptedBytes = File.ReadAllText(dataPath)
                                    .Split(',')
                                    .Select(byte.Parse)
                                    .ToArray();

            var allPossiblePasswords = from a in FirstChar.To(LastChar)
                                       from b in FirstChar.To(LastChar)
                                       from c in FirstChar.To(LastChar)
                                       select new[] { a, b, c };

            var possibleDecryptions = from trialPassword in allPossiblePasswords.AsParallel()
                                      let repeatedPassword = trialPassword.Repeat()
                                      select encryptedBytes
                                          .Xor(repeatedPassword)
                                          .ConvertAsciiBytesToString();

            var languageAnalyser = new LanguageAnalyser();

            var decryptionsWithDistanceToEnglish = from decryptedString in possibleDecryptions
                                                   select new
                                                              {
                                                                  DecryptedString = decryptedString,
                                                                  Distance = languageAnalyser.DetermineCloseness(decryptedString, LanguageProfiles.English)
                                                              };

            var mostLikelyDecryption = decryptionsWithDistanceToEnglish.MinItem(p => p.Distance);

            var sumOfAsciiValues = mostLikelyDecryption.DecryptedString
                .ConvertStringToAsciiBytes()
                .Aggregate(0, (runningTotal, next) => runningTotal += next);

            return sumOfAsciiValues;
        }
    }
}

Thursday, 10 June 2010

Practical Linq #5: Guessing the language of a piece of text

Google Chrome is magic: getting ready for my summer holiday to Brugge, I hit the city’s homepage. A little bar popped up at the top of the page: “This page is in Dutch. Would you like to translate it?”

image

How does Chrome know that? I suppose the domain name “.be” is one clue, but they speak French as well as Dutch in Belgium, so that by itself is not enough. Chrome does the same trick for pages written in a whole raft of languages, from Afrikaans to Yiddish.

I don’t know what Google’s secret sauce is, but I can show you one simple and surprisingly effective technique with an elegant implementation using LINQ in C#.

Studies in Linguistic Geometry

It is built around the observation that in each sufficiently large sample of text in a given language, letters of the alphabet will all occur with roughly the same frequencies. In English for example, the letter ‘e’ tops the popularity charts, making up about 13% of the average body of text. ‘t’ and ‘a’ are next, accounting for 9% and 8%. By contrast, in Portuguese, ‘a’ is the most frequent, at 15%, with ‘e’ and ‘o’ close behind.

So letter frequencies can be used as a kind of signature, or DNA profile, for a language. But given a sample of text, how do we decide which language signature it matches most closely? This is where we switch from linguistics to geometry.

In 2-dimensional space, we calculate the distance between two points using Pythagoras’ theorem. image The same thing works for two points in the real, 3-d, world. The distance from (x1, y1, z1) to (x2, y2, z2) is

image

But why stop there? What about the distance between two points in 26-dimensional space? You guessed it!

image

26-d space? Why bring that into the equation? Well think about those language signatures, giving the frequency of occurrence of each letter. You can reinterpret those signatures as points in 26-d space: the point representing the signature for English would be at 0.13 along the ‘e’ axis, 0.09 along the ‘t’ axis, 0.08 along the ‘a’ axis, and so on.

This then gives us our straightforward method of determining which language our sample text belongs to. We do a frequency analysis on its letters, then imagine the result of that analysis as a point in 26-d space, and work out which language signature it lies closest to using Pythagoras. Easy!

So how does it look in LINQ?

Linq to Linguistics

We start off with the frequency analysis (taking care to normalise the text to lower-case):

IEnumerable<CharacterFrequency> CalculateCharacterFrequencies(string sample)
{
    var characterFrequencies = sample
        .Select(char.ToLower)
        .GroupBy(c => c)
        .Select(group => new CharacterFrequency
        {
            Character = group.Key,
            Frequency = (double)group.Count() / sample.Length
        });

    return characterFrequencies;
}

Next we introduce a LanguageSignature class to hold the signature of each language:

[DebuggerDisplay("Language = { Language }")]
public class LanguageSignature
{
    private IDictionary<char, double> _frequencies;

    public LanguageSignature(string language, IDictionary<char, double> characterFrequencies)
    {
        Language = language;
        _frequencies = characterFrequencies;
    }

    public string Language { get; protected set; }

    public double GetFrequency(char character)
    {
        double frequency;
        return _frequencies.TryGetValue(character, out frequency) ? frequency : 0;
    }
}

Here are some snippets of sample signatures, data courtesy of Wikipedia

public static class LanguageSignatures
{
    public static LanguageSignature English =
        new LanguageSignature(
            "English",
            new Dictionary<char, double>
                {
                    {'a', 0.08167},
                    {'b', 0.01492},
                    {'c', 0.02782},
                    /// etc.
                });

    public static LanguageSignature French =
        new LanguageSignature(
            "French",
            new Dictionary<char, double>
                {
                    {'a', 0.07636},
                    {'b', 0.00901},
                    // etc.
                });

    
}

Now we do our sums in 26-d space, ignoring any letters that we don’t have frequency data for (note that Sqrt is a little extension method I created on the Double class that simply calls Math.Sqrt):

double CalculateDistanceFromSignature(LanguageSignature signature, IEnumerable<CharacterFrequency> characterFrequencies)
{
    var distance = characterFrequencies
        .Where(characterFrequency => signature.GetFrequency(characterFrequency.Character) > 0)
        .Select(characterFrequency
            => Math.Pow(characterFrequency.Frequency - signature.GetFrequency(characterFrequency.Character), 2))
        .Sum()
        .Sqrt();
    return distance;
}

Now we’re ready to determine which language our sample is closest to:

public LanguageSignature DetermineMostLikelyLanguage(string sample, IEnumerable<LanguageSignature> signatures)
{
    var characterFrequencies = CalculateCharacterFrequencies(sample);

    var closestLanguage = signatures.Select(signature => new
    {
        Language = signature,
        Distance = CalculateDistanceFromSignature(signature, characterFrequencies)
    })
    .MinItem(languageDistance => languageDistance.Distance);

    return closestLanguage.Language;
}

MinItem is an extension method I knocked up that returns the item from a sequence that yields the smallest value from the expression you supply (in this case, the smallest distance):

public static T MinItem<T, TCompare>(this IEnumerable<T> sequence, Func<T, TCompare> comparatorSelector) where TCompare : IComparable<TCompare>
{
    var minItem = sequence.Aggregate(
        sequence.First(), 
        (current, min) => comparatorSelector(current).CompareTo(comparatorSelector(min)) < 0 ? current : min);

    return minItem;
}

And we’re done.

Testing, Testing, Un, deux, trois

To prove it, I created a little test that analysed samples of texts in several languages taken from Project Gutenberg (the test uses MbUnit’s very handy XmlData feature to read the different test cases from my xml file).

public class LanguageAnalyserTests
{
    [Test]
    [XmlData("//Sample", FilePath = "ProblemData\\LanguageSamples.xml")]
    public void TestAnalysis([Bind("Text")]string text, [Bind("@Language")]string expectedLanguage)
    {
        var analyser = new LanguageAnalyser();
        var determinedLanguage = analyser.DetermineMostLikelyLanguage(
            text
            new[] { LanguageSignatures.English, LanguageSignatures.French, LanguageSignatures.German, LanguageSignatures.Dutch, LanguageSignatures.Spanish, LanguageSignatures.Italian });

        Assert.AreEqual(expectedLanguage, determinedLanguage.Language);
    }
}

In my test I’ve got 9 samples in German, English, French, Dutch, Spanish and Italian, extracted at random from the Gutenberg archives and each sample is correctly identified by my analyser. I also did a little experiment to see how much text it needed to correctly distinguish the samples. Limiting the samples to 600 characters caused some confusion between Spanish and English, but 700 characters was sufficient in my tests to enable the analyser to pick the right language for all samples.

A Challenge

I actually developed this LanguageAnalyser to help me solve a problem on Project Euler. Which one was it? Answers in the comments please, or by email, and bonus marks to anybody who can guess what my solution looks like. Check back Monday for the answer, and for full source code to today’s post.

Update: The answer's here, and the source code is now on MSDN Code Gallery

Tuesday, 6 January 2009

Project Euler 89: Converting to and from Roman Numerals

It would not surprise me at all to learn that the Roman Numeral system was invented by a Stone Mason with one eye on his retirement fund. As he charged per character chiseled he could expect a substantial fee whenever called upon to inscribe numbers; dates on tombstones were even better, because on the whole he could expect the length of the inscription and thus the fee to increase as the years progressed.

Everybody knows how the Roman Numeral system works, though the only time we usually get to show off our skills is when the end-credits roll on the the big or small screen and the date of film or programme production flashes up. It consists of 7 symbols, each standing for a value:

Numeral Value
I 1
V 5
X 10
L 50
C 100
D 500
M 1000

A number in Roman Numerals is just a string of these numerals written in descending order (e.g. M's first, followed by D's, etc.). To convert from Roman Numerals to decimal we just work through the string, adding up the values of the numerals. So MMVIIII represents 2 x 1000 + 1 x 5 + 4 x 1 = 2009.

The only complication is the subtractive rule. This rule makes it possible for some numbers to be written in a more compact form (I'm sure archaeologists will one day unearth a document recording the protests made by the Mason's Union on its introduction!). It allows a numeral of a lower value to be placed before one of a higher value to indicate that the lesser should be subtracted from the greater. Thus IV means 5 - 1 = 4, and XC = 100 - 10 = 90. Unfortunately, it's not quite that simple. According to PPI MMDCL1, only I, X and C can be used subtractively, and then only in front of numerals up to ten times as big. So I can only be used in front of V and X, X in front of L and C; and C in front of D and M.

Converting Numerals to Decimal Programmatically

It's the subtractive rule that stumped me at first when I tried to figure out an algorithm to parse roman numerals. It certainly makes things complicated if you think about parsing the string from left to right. But as soon as you start thinking backwards, that is working from right to left, the difficulty evaporates:

  1. Split the string into characters
  2. Convert each character into the value it represents
  3. Start from the right - the lowest value numeral
  4. Keep a running total, and a record of the maximum numeral encountered so far
  5. Take characters one by one:
    1. If character is greater than the maximum, add it to the running total and update the maximum
    2. If character is less than the maximum, subtract it from the running total

This converts straight-forwardly into LINQ:

public static int ConvertToDecimal(this string numerals)
{
    var result = numerals
        .ToCharArray()
        .Reverse()
        .Select(c => _numeralToNumberLookup[c.ToString()])
        .Aggregate(
            new { MaxValue = 0, RunningTotal = 0 },
            (state, item) => new
            {
                MaxValue = Math.Max(state.MaxValue, item),
                RunningTotal = item >= state.MaxValue ? state.RunningTotal + item 
                                                               : state.RunningTotal - item
            }, 
            aggregate => aggregate.RunningTotal);

    return result;
}

You'll note that I'm using an anonymous type in my call to Aggregate to allow me to record both pieces of state. Notice that I have to seed the call to Aggregate with an empty instance of the anonymous type (line 8). The C# compiler makes this possible by ensuring that any anonymous type that has properties with the same name and type and in the same order as another anonymous type actually become the same type. This overload of Aggregate also requires a lambda expression that extracts the overall result from the final aggregate object.

Converting Numbers to Numerals (with added recurserators!)

So that's Parsing taken care of. What about turning a number into numerals?

The algorithm isn't much more difficult, though the implementation I've chosen is a little unusual. The algorithm is recursive. We start off with the number we want to convert, and a stack containing the values of all the numerals that are available to us, greatest on the top. This stack includes the values of "composite numerals" - pairs of numerals that follow the subtractive rule, for example, 4 (IV), 9 (IX) and 40 (XL). In each iteration we peek at the numeral value on the top of the stack and test to see whether the number is divisible by it. If it is we've got us some numerals - we produce part of the final numeral string by looking up the appropriate numeral(s) and repeating them as many times as the number is divisible by the numeral value. If not, we recurse anyway. To recurse, we pop the top numeral off the stack, and pass the reduced list of numerals through to the next iteration, along with the remainder of the number after dividing by the numeral value. We terminate the recursion when we are asked to convert number 0 - which can't be done in roman numerals. If you understood all that, you have permission to skip the next paragraph; otherwise read on for a worked run-through of the algorithm.

For example, suppose we want to convert 2051. We start with the complete list of numerals: { 1000, 900, 500, 400, ... 50, 40, 10, 9, 1}. Peek at the first numeral value, and discover that it's 1000. 2051 is 2 x 1000, with remainder 51. Looking up 1000 in our numeral dictionary yields "M", so the first part of the numeral string is "MM". We're done with that iteration, so we remove 1000 from our list of numerals, and pass the smaller list, and the remainder 51 to the next iteration. The next few numeral values {900, 500, etc.} don't divide into 51, so we keep popping numerals, passing remainders and recursing till we get to a numeral list starting {50, 40, ...}.  51 is divisible by 50 (so we deduce that the next part of the numeral string is "L"). We then pop, pass and recurse until the number 1 is given for conversion, and the numeral list just contains {1}. At this point we add numeral "I" to the result, recursing hits the terminating condition, so we stop.

Now for the implementation. I could have used a straight-forward recursive function: I did this with my solution to Problem 31 (Counting Coin Combinations) which had a similar algorithm. But my Lazy Trampoline hasn't seen much action recently, so I decided to give it a bounce. The idea of the Lazy Trampoline is to create a recursive function that is able to yield a result after each iteration - a recurserator, you could call it.

It looks like this2:

public static string ConvertToNumerals(this int number)
{
    Func<int, IStack<int>, IEnumerable<string>> numeralsEnumerator = Trampoline.MakeLazyTrampoline((int numberToConvert, IStack<int> availableNumerals) =>
    {
        if (numberToConvert == 0)
        {
            return Trampoline.YieldBreak<int, IStack<int>, string>();
        }

        int currentNumeral = availableNumerals.Peek();

        int quotient = numberToConvert / currentNumeral;
        int remainder = numberToConvert % currentNumeral;

        string numberAsNumerals = _numberToNumeralLookup[currentNumeral].Repeat(quotient);

        return Trampoline.YieldAndRecurse<int, IStack<int>, string>(
            numberAsNumerals, 
            remainder, availableNumerals.Pop());
    }
    );

    var result = numeralsEnumerator(number, _availableNumerals)
             .Aggregate(
                new StringBuilder(),
                (sb, numerals) => sb.Append(numerals),
                sb => sb.ToString());

    return result;
}

The call to MakeLazyTrampoline (line 3) generates an iterator (numeralsEnumerator). Each item in the sequence generated by the iterator is a string consisting of one type of numeral. These are all combined by the query starting line 23 to create the complete numeral.

Within the recurserator, note that I return special values generated by Trampoline.YieldBreak and Trampoline.YieldAndRecurse. These control the behaviour of the recurserator. YieldBreak indicates that the recursion should stop now; YieldAndRecurse indicates (as its first parameter) which value should be returned in the sequence, and (in the second and third parameters) which values should be passed through to the next iteration.

It may have slipped under you radar, but in Line 15 I'm making a call to the Repeat method which doesn't exist on the String class. Here's the missing extension method definition:

public static string Repeat(this string value, int count)
{
    if (count == 0)
    {
        return string.Empty;
    }
    
    if (count == 1)
    {
        return value;
    }

    return Enumerable.Repeat(value, count)
        .Aggregate(new StringBuilder(), 
        (aggregate, item) => aggregate.Append(item), 
        aggregate => aggregate.ToString());
}

Solving Project Euler Problem 89

Project Euler Problem 89 gives us a file containing a load of numbers in Roman Numeral form, and asks us to calculate how many characters can be saved if the numbers are compacted using the subtractive rule. Given the two extension methods we've just implemented, the solution is a simple pair of LINQ queries:

[EulerProblem(89, Title = "Develop a method to express Roman numerals in minimal form.")]
class Problem89
{
    public long Solve()
    {
        var lines = File.ReadAllLines("ProblemData\\Problem89Data.txt");

        var numberOfCharacters = lines.Select(line => line.Length).Sum();

        var minimisedNumberOfCharacters = lines
            .Select(line => line.ConvertToDecimal())
            .Select(number => number.ConvertToNumerals())
            .Select(numerals => numerals.Length)
            .Sum();

        return numberOfCharacters - minimisedNumberOfCharacters;
    }
}

The full code is available, as always, on the MSDN Code Gallery page. Hope you like it Saevar.

Footnotes
  1. PPI - Prex pro Ineo. ie Request For Comment! Thanks, InterTran!
  2. Thanks again to Eric Lippert for his Immutable Stack code.

Thursday, 16 October 2008

Project Euler Code Clearance

Roll up! Roll up! Get them while they're hot, don't leave them 'till they're not. Get your genuine, authentic Functional Fun Project Euler solutions here. Bargain prices, can't beat 'em. Visual Studio solution thrown in free.

I've been pretty busy over the last week or so, and it looks like I'll continue to be busy until I depart for LA at the end of next week. And you know what that means: .Net 4, C# 4, Oslo, Cloud Computing, Windows 7. That gang of acronyms and code names should keep me in blog posts for a good long while. Meanwhile, I have several Project Euler solutions languishing in the basement of my hard-disk (fourth platter, third sector, last block on the right, I think), and I can't see myself getting the time to do them justice in individual blog posts.

So today I'm going to do them an injustice and launch them into the world with barely a scant mention each. As usual, you can get the code from MSDN Code Gallery.

Problem 20: Summing the digits of a Factorial

Problem 20 gave me a chance to reuse my code for summing large numbers again. That's because calculating a factorial can be done iteratively. Imagine the sequence 1!, 2!, 3!, etc. You get the (n + 1)th term by multiplying the nth term by itself n + 1 times (that's just the definition of factorial). And the multiplication boils down to adding up n + 1 copies of the nth term. Wrap that simple idea inside an application of Unfold, and you have the solution.

Problem 22: Number crunching names

Problem 22 is just a file processing problem. Nothing more to it than that. It has a nice, one line solution with LINQ though:

return File.ReadAllText(problemDataFileName)
           .Split(new[] {','})
           .Select(name => name.Trim(new[] {'\"'}))
           .OrderBy(name => name)
           .Select((name, index) => (index + 1) * name.ToCharArray()
													  .Select(letter => (letter - 'A') +1)
                                                      .Sum())
           .Sum();

Problem 25: Big Fibonacci numbers

In Problem 25, Euler wants to know the first term in the Fibonacci sequence that has 1000 digits, which is of course his way of getting us to find a way of computing with big numbers. No problem to us though: we've got the Unfold method to calculate the Fibonacci sequence - and did I mention the code I wrote before that we can use to sum the big numbers we'll find?

Problem 31: Counting Coin Combinations

I'm quite pleased with my solution to Problem 31, wherein Euler asks us to count the number of ways in which £2 can be given using the British coinage system. I came up with a recursive algorithm that starts with the value of change you wish to give and a list of available coin types. Then it removes the biggest coin from the list, works out how many times that coin could be used, and for each possibility calculates the number of combinations for the reduced amount, and the reduced coin set by calling itself recursively.

For example, if we needed to make £1 using 10p and 2p, and 1p coins, we'd start by seeing that we could use between zero and ten 10p coins, so there are eleven possibilities we need to investigate. For each possibility we calculate how much remains once we've put down the appropriate number of 10 pences, then use the same algorithm again, but considering just 2p and 1p coins.

Problem 36: Binary and Decimal Palindromes

We last encountered numeric palindromes in Problem 4: they're numbers that read the same in both directions. In Problem 4 we were only interested in numbers that are palindromic when written in base 10. Problem 36 asks us to count the numbers that are palindromic when written in binary and in decimal. The most interesting part about this problem was finding a number's representation in binary. I could probably have used the BitArray class to do this, but instead chose to use this LINQ query:

private static IEnumerable<bool> GetNumberBits(uint number)
{
    if (number == 0)
    {
        return new[] {false};
    }

    // iterate through the bit positions, checking whether that
    // bit of the the number is set;
    // do it in reverse, so that we can ignore leading zeros
    return 31.To(0)
        .Select(bit => (number & (1 << bit)) == (1 << bit))
        .SkipWhile(bit => !bit);
}

Problem 112: Finding the density of Bouncy numbers

Mathematicians don't try to hide their obsession with numbers, do they? They make it plain as day that they count numbers amongst their closest friends, by giving them all names. It's the Bouncy numbers that are the heroes of Problem 112. These are numbers that, considering their digits as a sequence have a neither increasing nor decreasing sequence. My solution to this problem puts the Aggregate and LazilyAggregate methods to a rather unconventional use.

Friday, 8 August 2008

Project Euler Problem 17: Converting numbers to words

Zimbabwe Prices: Picture Credit, SokwaneleFancy earning a Lotto prize of 1.2 quadrillion dollars? Then you should move to Zimbabwe! Don't be in too much of a hurry, though. By the time you'd brought it back home, your Zimbabwe-dollar haul would have shrunk to less than 4,000 US Dollars.

A couple of weeks back the BBC published an article talking about the problem of big numbers in Zimbabwe. Due to daily expenses measured in the trillions of dollars, with hyperinflation of 2,200,000% pushing that figure ever higher, the citizens of Zimbabwe have been googling to find the names for the next big numbers after trillions. Coincidentally, I was doing the same thing the day before I read the article as I went about solving Project Euler Problem 17.

The problem wants to know how many letters are needed to write out the numbers 1 to 1000 in words. Clearly this calls for an algorithm to convert numbers to words; since I always like to give a little extra to my loyal readers, I wanted to create an algorithm to handle numbers bigger than 1000. I ended up with one that can convert anything up to long.MaxValue: in words, nine quintillion, two hundred and twenty-three quadrillion, three hundred and seventy-two trillion, thirty-six billion, eight hundred and fifty-four million, seven hundred and seventy-five thousand, eight hundred and seven! Plenty powerful enough to write out Bill Gate's paychecks, and sufficient to keep the Zimbabweans going for the next few months!

So how does the algorithm work? A basic ingredient is a dictionary mapping the basic numbers to words. This covers the numbers 1 to 20, then every multiple of ten up to 100, then each larger number that has a special name. The key ingredient is a recursive function (functional programming had to come into it somewhere) that starts with the largest part of a number and repeatedly breaks it down into its components.

In Pseudo-code:

  1. Let n be the number to process
  2. For numbers smaller than 0, combine "Negative" with the result of calling the algorithm on the absolute value of n
  3. For n between 0 and 20, look up the result in the word dictionary
  4. For n between 20 and 100, calculate the number of 10s and number of units; look up the name for the number of tens; if there are any units, look up the appropriate name and combine the two results with a hyphen
  5. For n between 100 and 1000, calculate the number of 100s and the remainder; look up the name for the number of hundreds; if there is any remainder, recurse to get its wordy representation, then combine it with result for the hundreds using "and" to join the two parts
  6. For n bigger than 1000: decide which is the biggest named number (million, billion, etc.) that divides into n. Calculate the number of that base unit and the remainder. Recurse to convert the number of base units into words, and recurse to convert the remainder into words. Combine the two parts using ","

In C#:

public static class ConvertToWordsExtension
{
 private static Dictionary<long, string> _wordDictionary;
 private const int OneThousand = 1000;
 private const long OneMillion = 1000000;
 private const long OneBillion = 1000000000;
 private const long OneTrillion = 1000000000000;
 private const long OneQuadrillion = 1000000000000000;
 private const long OneQuintillion = 1000000000000000000;

 /// <summary>
 /// Converts a number to its English representation in words.
 /// </summary>
 /// <remarks>Uses the Short Scale for large numbers. See http://en.wikipedia.org/wiki/Long_and_short_scales for more details</remarks>
 public static string ConvertToWords(this long number)
 {
     EnsureWordDictionaryInitialised();

     if (number == long.MinValue)
     {
        throw new ArgumentOutOfRangeException();
     }

     return ConvertToWordsCore(number);
 }

 /// <summary>
 /// Converts a number to its English representation in words
 /// </summary>
 public static string ConvertToWords(this int number)
 {
     return ConvertToWords((long)number);
 }

 private static Dictionary<long, string> CreateWordDictionary()
 {
     return new Dictionary<long, string>
                {
                    {0, "zero"},
                    {1, "one"},
                    {2, "two"},
                    {3, "three"},
                    {4, "four"},
                    {5, "five"},
                    {6, "six"},
                    {7, "seven"},
                    {8, "eight"},
                    {9, "nine"},
                    {10, "ten"},
                    {11, "eleven"},
                    {12, "twelve"},
                    {13, "thirteen"},
                    {14, "fourteen"},
                    {15, "fifteen"},
                    {16, "sixteen"},
                    {17, "seventeen"},
                    {18, "eighteen"},
                    {19, "nineteen"},
                    {20, "twenty"},
                    {30, "thirty"},
                    {40, "forty"},
                    {50, "fifty"},
                    {60, "sixty"},
                    {70, "seventy"},
                    {80, "eighty"},
                    {90, "ninety"},
                    {100, "hundred"},
                    {OneThousand, "thousand"},
                    {OneMillion, "million"},
                    {OneBillion, "billion"},
                    {OneTrillion, "trillion"},
                    {OneQuadrillion, "quadrillion"},
                    {OneQuintillion, "quintillion"}
                };
 }

 private static void EnsureWordDictionaryInitialised()
 {
     // ensure thread-safety when caching our word dictionary
     // note: this doesn't prevent two copies of the word dictionary
     // being initialised - but that doesn't matter; only one would be
     // cached, the other garbage collected.
     if (_wordDictionary == null)
     {
         var dictionary = CreateWordDictionary();
         Interlocked.CompareExchange(ref _wordDictionary, dictionary, null);
     }
 }

 private static string ConvertToWordsCore(long number)
 {
     if (number < 0)
     {
         return "negative " + ConvertToWordsCore(Math.Abs(number));
     }

     // if between 1 and 19, convert to word
     if (0 <= number && number < 20)
     {
         return _wordDictionary[number];
     }

     // if between 20 and 99, convert tens to word then recurse for units
     if (20 <= number && number < 100)
     {
         return ProcessTens(number, _wordDictionary);
     }

     // if between 100 and 999, convert hundreds to word then recurse for tens
     if (100 <= number && number < OneThousand)
     {
         return ProcessHundreds(number, _wordDictionary);
     }

     if (OneThousand <= number && number < OneMillion)
     {
         return ProcessLargeNumber(number, OneThousand, _wordDictionary);
     }

     if (OneMillion <= number && number < OneBillion)
     {
         return ProcessLargeNumber(number, OneMillion, _wordDictionary);
     }

     if (OneBillion <= number && number < OneTrillion)
     {
         return ProcessLargeNumber(number, OneBillion, _wordDictionary);
     }

     if (OneTrillion <= number && number < OneQuadrillion)
     {
         return ProcessLargeNumber(number, OneTrillion, _wordDictionary);
     }

     if (OneQuadrillion <= number && number < OneQuintillion)
     {
         return ProcessLargeNumber(number, OneQuadrillion, _wordDictionary);
     }
     else
     {
         return ProcessLargeNumber(number, OneQuintillion, _wordDictionary);
     }
 }

 private static string ProcessLargeNumber(long number, long baseUnit, Dictionary<long, string> wordDictionary)
 {
     // split the number into number of baseUnits (thousands, millions, etc.)
     // and the remainder
     var numberOfBaseUnits = number / baseUnit;
     var remainder = number % baseUnit;
     // apply ConvertToWordsCore to represent the number of baseUnits as words
     string conversion = ConvertToWordsCore(numberOfBaseUnits) + " " + wordDictionary[baseUnit];
     // recurse for any remainder
                 conversion += remainder <= 0 ? ""
                            : (remainder < 100 ? " and " : ", ") + ConvertToWordsCore(remainder);
     return conversion;
 }

 private static string ProcessHundreds(long number, Dictionary<long, string> wordDictionary)
 {
     var hundreds = number / 100;
     var remainder = number % 100;
     string conversion = wordDictionary[hundreds] + " " + wordDictionary[100];
     conversion += remainder > 0 ? " and " + ConvertToWordsCore(remainder) : "";
     return conversion;
 }

 private static string ProcessTens(long number, Dictionary<long, string> wordDictionary)
 {
     Debug.Assert(0 <= number && number < 100);

     // split the number into the number of tens and the number of units,
     // so that words for both can be looked up independantly
     var tens = (number / 10) * 10;
     var units = number % 10;
     string conversion = wordDictionary[tens];
     conversion += units > 0 ? "-" + wordDictionary[units] : "";
     return conversion;
 }
}

Having coded the algorithm like that, I was casting around to find a way of making the ConvertToWordsCore method more compact. That was when I came across Igor Ostrovsky's post on the ternary conditional operation ("?") and was reminded of a neat trick. I re-wrote the method like this:

private static string ConvertToWordsCore(long number)
{
 return
     number < 0 ? "negative " + ConvertToWordsCore(Math.Abs(number))
         : 0 <= number && number < 20 ? _wordDictionary[number]
         : 20 <= number && number < 100 ? ProcessTens(number, _wordDictionary)
         : 100 <= number && number < OneThousand ? ProcessHundreds(number, _wordDictionary)
         : OneThousand <= number && number < OneMillion ? ProcessLargeNumber(number, OneThousand, _wordDictionary)
         : OneMillion <= number && number < OneBillion ? ProcessLargeNumber(number, OneMillion, _wordDictionary)
         : OneBillion <= number && number < OneTrillion ? ProcessLargeNumber(number, OneBillion, _wordDictionary)
         : OneTrillion <= number && number < OneQuadrillion ? ProcessLargeNumber(number, OneTrillion, _wordDictionary)
         : OneQuadrillion <= number && number < OneQuintillion ? ProcessLargeNumber(number, OneQuadrillion, _wordDictionary)
         : ProcessLargeNumber(number, OneQuintillion, _wordDictionary); // long.Max value is just over nine quintillion
}

That's much more compact, and to my eyes, a lot more elegant. The only downside I can see is that debugging might be more difficult. Which form do you prefer?

With this algorithm in place, solving Project Euler 17 is trivial:

[EulerProblem(17, Title="How many letters would be needed to write all the numbers in words from 1 to 1000?")]
public class Problem17
{
 public long Solve()
 {
     return
         1.To(1000)
             .Select(
                 number => number
                             .ConvertToWords()
                             .ToCharArray()
                             .Count(character => char.IsLetter(character))
             )
             .Sum();
 }
}

As always, full code (including some unit tests for the ConvertToWords method!) is available on my Project Euler code gallery page. You can also try out the algorithm in my handy online utility to convert numbers to words.

Update 27/5/09:Fixed bug where 'and' was missed out in numbers like 1006