From C# To Perl: Performance

Caveats

Let’s talk about the performance of C# vs. Perl. First, let me just say that a performance comparison of a staticly-typed, pre-compiled language vs. a dynamically-typed, scripting language is inherently unfair. Yes, .NET does not compile to native code, but the comparison is still unfair. A static, type-safe language avoids the overhead of compilation on startup and allows for more optimization, particularly removing virtual calls when not needed.

Comparison

With that being said, how much faster is C# in benchmarks? My usual source when comparing the performance of programming languages is the The Computer Language Benchmarks Game. Here you can see the comparison of C# vs. Perl. One thing to note is that I am comparing the performance of the Mono implementation of C# vs. Perl, not the Microsoft .NET implementation. The performance of Mono is comparable to that of .NET these days and the benchmark game runs the benchmarks on Ubuntu, so I’m just going to run with it. Also, while small benchmarks have severe limitations, they will serve the purpose here.

We can see three metrics in the comparison: time, memory, and code size. Various benchmarks stress each differently, but the trends are clear. First, C# is a hell of a lot faster than Perl. As I stated above, this shouldn’t surprise anybody.

The more interesting comparisons are memory and code. We can see that Mono uses 13x more memory on the mandelbrot benchmark, but is anywhere from even to 3x more on every other. It also requires up to 5x more code than Perl. You can see the source code for the benchmarks that required much more code in C# below.

k-nucleotide: C# Perl
reverse-complement: C# Perl

You can see that the code for the k-nucelotide benchmark is much more concise in Perl. The verboseness of C# in this case hinders the ablity to quickly understand the algorithm. The same is true for reverse-complement, though not quite as bad, with Perl’s concise file-handling operations being the main difference.

You can also see the same comparison made on a quad-core 32-bit machine. The same trend holds and you can also see that Mono makes better use of multiple cores. Unfortunately, benchmarks are not available for Perl on 64-bit processors.

Does It Matter

So what? C# is faster. How often does it really matter? If you aren’t doing heavy number crunching, it probably doesn’t. These days performance is often limited by outside factors, such as network latency or database performance. But let’s say you have identified a bottleneck in your code and it is definitely your code that is the bottleneck. How would you go about improving performance in each language?

You have similar options for both languages. In C# you can link to native code (usually a compiled C/C++ binary) or you can declare your code to be unsafe to get direct access to pointers. Either way, it’s going to introduce some pain. Integrating C# with native code always made me feel dirty and I usually ended up putting a wrapper class around it to hide the ugliness. Declaring your code to be unsafe isn’t much better. Most of its speed benefits come from getting the runtime out of the way, meaning you lose most of the benefits of C# over C/C++.

Perl gives you a few options that are mostly equivalent to those in C#. First, you can inline C code with the Inline module. This is admittedly pretty sweet for simple cases. You don’t have to use C, but if you are doing it for performance reasons you probably will. This is great if you have a single performance sensitive algorithm in your module, because, as the name implies, you can do it inline. However, there are limitations to what you can do with this method. If you need to improve the performance of a whole module, you are best served by using XS. This is not for the faint at heart, but will allow you to transparently call C code within Perl.

Conclusion

What we have found is pretty much what was expected. C# is much faster, but uses more memory than Perl and is more verbose. They both have similar options to improve performance by integrating with code written in C, although Perl’s seem slightly more natural to me.

Once again though, does it really matter? Most programmers are not tackling performance sensitive problems and hardware is cheap. If performance does matter, you’re probably better off with C#, but in the grand scheme of things, whatever tools your team is familiar with is the far more important factor.

8 Responses to “From C# To Perl: Performance”

  1. Hugh S. Myers

    Given what programmers and their time cost, it may be cheaper from here on out to always throw hardware at the problem. Another factor is the decreasing information available to those working on problems that might need optimization; i.e. those who have the ability to switch as needed to something like assembler are almost extinct— most certainly on the decrease…

  2. Isaac Gouy

    >> programming language shootout

    Please correct the text – although the correct name is “The Computer Language Benchmarks Game” for short just write “benchmarks game”.

    (The Virginia Tech shooting in April 2007 once again pushed gun violence into the media headlines. There was no wish to be associated with or to trivialise the slaughter behind the phrase shootout so the project was renamed back on 20th April 2007.)

  3. Shimshon

    I wrote an email engine in pure Perl (but using the epoll syscall) that could send 150K emails per second sustained. That includes DNS (which I cached) lookups and SMTP session. I’m pretty sure I could have gotten it to 200K.

    So while I have no doubt that C# and other similar languages can be much faster, the performance I’ve been able to get from Perl for most tasks is surprisingly high and sufficient (as network bandwidth is pretty much saturated). Plus, there’s the fact that Perl is just so much more expressive.

  4. Brian Meeker

    @Isaac: Updated the language. I didn’t notice the name change. And thanks for the .NET vs. Mono comparison. It pretty much confirmed what I had read anecdotally. I guess I should have read the help page.

    @Shimshon: Perl performance is definitely sufficient for most tasks. I would disagree with you about Perl’s expressiveness though to some extent. C# 4.0 is a very expressive language. It’s come a long way from just being a rehash of Java. My experience has been that scripting languages tend to be more readable in small programs, but require more discipline for larger ones to achieve the same clarity as C#.

  5. Isaac Gouy

    Thanks for correcting the website name!

    >> We can see that Mono uses 13x more memory on the mandelbrot benchmark

    We can also see that very same C# mandelbrot program can run in 1/4 of the time on quad-core with that same memory use (38,808KB), but the Perl program that uses quad-core takes 7.6x more memory (294,844KB) than the C# program :-)

    >> It also requires up to 5x more code than Perl.

    Also we can see that one C# program written to exploit multi core uses 2x more code than a naive C# program that can only use one core.

    Also we can see that one Perl program written to exploit multi core uses 2x more code than a naive Perl program that can only use one core.

    Be careful not compare the length of C# programs written to exploit multi core against the length of naive Perl programs that can only use one core :-)

    (It isn’t that it “requires up to 5x more code than Perl”, shorter slower C# programs have probably been discarded – the benchmarks game keeps the fastest and weeds out the slower programs.)

    • Brian Meeker

      Good catch. I was planning to do another post comparing the parallel programming models of the two languages so I glossed over that. I haven’t done any in Perl yet though.

      • Isaac Gouy

        1) Just keep in mind that the emphasis in the benchmarks game is on time rather than memory use, time rather than code size.

        Maybe we’d see something different if we said we were going to rank the programs according to which used the least memory or which used the least code?

        Best to think of the benchmarks game as having time as primary, and memory use and code size measures being distant tertiary criteria – so for the fastest programs we might bring those other criteria into the evaluation.

        2) And there are different reasons for caring about memory use – we want the smallest memory use for programs that don’t allocate much memory, or we want the smallest memory use for programs that need to allocate a lot of memory, or …

        Like other VM implementations Mono C# has a (somewhat ~5MB) high default memory use, but that doesn’t mean programs that need to allocate a lot of memory will have unusually high memory use. For example, binary-trees, k-nucleotide, reverse-complement, regex-dna.