Category Archives: .NET

Storing Time Series in Redis

Last time, I explored how to store time series in Microsoft Azure Table Service. This time I’ll do the same but in Redis. It’s is a very popular key-value store (but not only) and I highly encourage you to review it if you still don’t know it. In addition to my latest post it’s important to note that Time Series have very special properties.

  • Append only
    Most of the time, collectors or emitters append data “at the end”.
  • Readonly mode
    Time Series are never updated. Points are just read to create beautiful charts, compute statistics, etc … Most probably, latest points are more important than the others : it’s common to use last minute, last hour, last day in charts.
  • Big Data
    A simple collector that runs every second creates 86 400 points per day,  2 592 000 per months, 31 536 000 per year… and this is just for a single metric.

There are many ways to implement Time series in Redis, for example by using sorted sets, lists or even hashes. None is really better and the choice depends on your context (yes, again). A few weeks ago, for one of my needs, I needed a very efficient way to store thousands of thousands of data points.  Here are the details of this implementation, heavily inspired by a reposity of @antirez https://github.com/antirez/redis-timeseries

How it works

The main idea is to use the APPEND command.

If key already exists and is a string, this command appends the value at the end of the string. If key does not exist it is created and set as an empty string, so APPEND will be similar to SET in this special case.

redistsdatapoints
Every time series is stored into a single string containing multiple data points that are mostly ordered in time. So, the library appends every data point as a serialized string terminated by a special character (#).Internally, every data point contains two parts: a timestamp and a value (also delimited by a special character (‘:’).

Now to make things a little bit more efficient, Key Names are generated via a basic algorithm. The key name contains a constant (user-defined) and a rounded timestamp, so we will not add all the points to the same key. The rounding factor depends on your configuration settings. If you set it to one hour, all data points inserted between 4PM and 5PM will go to the same key.

Reading data is done via GETRANGE. Why not a basic GET ? It’s mainly because each key can have a lot of data points. I don’t want to risk an OutOfMemoryException, Excessive Large Heap Allocations or GC … Depending of the rounding factor, it is also possible to hit redis several times. For example, if the rounding factor is set to one hour and you want the last 4 hours.

Pros Cons
Very space efficient Inserts at the end only
Fast inserts (APPEND is O(1)) Unordered list
Up to 512 MB Range queries in a serie not supported (requires fixed-size)
TTL on keys

Introducing RedisTS (Redis Time Series)

I’ve committed an implementation on github and a nuget package is available here. The usage is pretty trivial and you can inspect the test project or samples to understand how to use it.

Note : RedisTS depends on StackExchange.Redis. An open ConnectionMultiplexer is required and redists will never open a new connection or close the current.

At first, you have to create a client you’re your time series.

var options= new TimeSeriesOptions(3600 * 1000, 1, TimeSpan.FromDays(1));
var client = TimeSeriesFactory.New(db, "msts", options);

This operations is cheap and the client is –more or less- stateless. Once you have a client, you can add one or more datapoints. This is typically done via a background task or a scheduled task.

//here I add new datapoint. Date : now and Value : 123456789
client.AddAsync(DateTime.UtcNow, 123456789);

That’s all on client side. Now if you insert data points from one location, it’s expected to read them from another one, maybe to display beautiful charts or compute statistics. For example, if you want to get all the data points since one hour for the serie “myts”, you can use the following piece of code :

client.RangeAsync(DateTime.UtcNow.AddHours(-1), DateTime.UtcNow);

That’s all for today. I hope this will help you and feedback is heavily encouraged. Thanks

Moving to xUnit.net

xUnit.net is a free, open source, community-focused unit testing tool for the .NET Framework. Written by the original inventor of NUnit v2, xUnit.net is the latest technology for unit testing C#, F#, VB.NET and other .NET languages. xUnit.net works with ReSharper, CodeRush, TestDriven.NET and Xamarin.”

Many Test Framework are available in .Net : MS Test, NUnit, xUnit, … Testing is an important aspect in Agile and XP and you can’t reach Continuous Deployement without it. Today, we write more and more tests, sometimes we have more tests than production code ; that’s why a decent test framework is so important today.

Why another test framework?

There are several articles & posts on internet explaining the pros and cons of each Test Framework. Here are my –unordered- personal reasons:

  • Nuget ‘All the way’

You don’t need to install a vsix extensions to VS, or manually add a reference to a file-based dependency (Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll…. v10.0 or v10.1?)  or to create a specific Test Project. In xUnit, you just need a basic class Library with a reference to the official Nuget. The VS Test Runner is even a nuget. Everything is distributed via nupkg and it’s terribly simple and efficient.

  • Great community & active development

xUnit.net is free and open source. The code is hosted on github (800 commits, 30 contibutors)  and was previously on codeplex. The core team is made of inspired evangelists and very active. The official twitter account has 1400 followers and it’s still growing. xUnit is extensible and a lot of extensions like AutoFixture are already available on nuget.org.

  • Part of the next big thing

Do you know ASP.NET 5, Xamarin, DNX, …? xUnit is already compatible with all of these platforms. xUnit may be the first class citizen and default choice in .net in the future. It’s not useless to spend a couple of hours to understand it.

xunitplatforms

  • Well integrated in .NET ecosystem

There are troubles to switch to xUnit because it is already well integrated in the .net ecosystem: Team Foundation Server, CruiseControl.net, AppVeyor, TeamCity, Resharper, TestDriven.net, …

  • It helps to write “Better, Faster, Stronger” Tests

It’s maybe the most important reason. I have the impression that parameterized tests (Via Theories), Generic Tests, extensibility via Attributes, Shared Context, … remove a lot of glue and frictions in my test project. Assertions & Attributes are quite common compared to the others test Framework but there are a few additional interesting properties.xunittest

Integrating xUnit with Team Foundation Server

As I said, xUnit.net is already well integrated with several Build Server like AppVeyor or TeamCity. Using it with TFS is not so complicated but we will have a few extra configuration steps.

  1. Download the latest stable version of runner.visualstudio
  2. Change the extension to zip and extract all the files.
  3. 3 assemblies should be in the folder build\_common
  4. The 3 files show to committed somewhere in TFS (up to you)
  5. xunitdllsOpen Team Explorer Tab, Build -> Actions ->Manage Build Controllers, Select Properties. You should fill the TFS path to custom assemblies (xUnit)

controllersettings

That’s all. You can now write your tests and build your solution on the build agent. xUnit will be automatically used.

One particular problem in the current version is that TestFilters/TestCategories are not supported. xUnit has Traits but these ones are not understood by the runner. It’s common to mix unit tests & integration tests in a solution in order to have a better and stronger test strategy.

ctegories

If you are in this case, you have only two options:

  • Wait for xunit.runner.visualstudio 2.1 (summer 2015 ?)
    A recent commit added support to Test Filters. It’s a question a week to see it on prod. (Note : the fix is available on the public MyGet feed)
  • Create another solution without integration tests
    A little bit painful to maintain but having a clean and fast solution to build is a good practice.

I’m completely convinced by xUnit.net and I hope you too ! If you still don’t know it,  please give it a try.