Feb 28 2008

Zenoss System Monitoring

Tag: Open SourceSymon Rottem @ 2:10 pm

At the moment I’m working on a system that consists of about 45 machines working together, providing a range of services, all of which need to be monitored for availability and that everything is running within measured thresholds. It’s important that if any of these machines or services fail that someone is notified and that if the issue isn’t resolved in a reasonable time frame that some kind of escalation process will ensure others are notified until something is done.

Having no available budget when I began investigating options I started looking at FR/OSS solutions to solve the problem. After a couple of false starts with Nagios and Hyperic HQ (which only missed the cut because it’s free version was missing one particular feature I needed; the ability to schedule repetitive maintenance periods – a pity that because otherwise it looks like an excellent FR/OSS product) I took a look at Zenoss Core and had a nice surprise.

Zenoss does a spectacular job of simplifying the process of adding new machines to be monitored – once the hosts have been SNMP enabled (and WMI enabled, for Windows boxes) you can set up a network in Zenoss and tell it to scan the network for new devices after which it will dutifully add all detected hosts with SNMP.

What’s really nice is that you can simply switch devices from a basic detected profile type to a more specific type (ie, Windows host, Linux host, router, switch, etc.) and Zenoss will do further investigation on the device based on its type. For example, if you designate the device as a Windows host it will query for other Windows related information including the software and services installed on the box. Similarly if you select Linux or Solaris it will perform other OS specific checks, etc.

Also interesting is that it establishes an inventory of the software installed on all the devices so you can determine which machines are running which software. By default each host is re-profiled every 6 hours and if any changes are detected the database is updated and you can be notified of the changes.

Once the hosts have been added Zenoss dutifully harvests issues from system logs, checks for availability of designated processes or services, and tracks values like available memory and processor usage (and yes, even custom data can be collected) over time. Hell, there are event pretty graphs for you to look at. Once the data is coming in notifications can be configured which can be triggered by outages or data exceeding thresholds and they can be sent by a variety of methods (email is one and is certainly the easiest to get running, but there are SMS/paging options amongst others).

Zenoss isn’t perfect, however. It can perform very slowly sometimes due to the way it manages caching data – it looks like it uses all available system memory to the point where it actually uses up the swap space as well. And navigation can sometimes be a pain as you have to move through multiple menus to get to sub groupings of machines unless you’re prepared to type the url to the group by hand.

Regardless, overall I’ve been pretty impressed with what Zenoss can do. There are some features missing from Zenoss Core, but their enterprise version seems to address most of these, and since it’s OSS there’s nothing you can’t choose to do yourself…if you can find the time.

Feb 27 2008

More Configuring NHibernate Caches

Tag: NHibernate,ORMSymon Rottem @ 6:52 am

One of my readers recently asked to see a sample of configuring NHibernate caching through the config file after reading my previous post Configuring NHibernate Caches, so here goes. Please bear with me in case there are any typos as this has been rolled by hand.

In this particular example I’m configuring NHibernate to use SysCache for the second level cache and have done it all in the app.config (or web.config if it’s a web application). Note that this is not a fully complete config file but deals with the parts necessary for this example.


<section name="hibernate-configuration" type="NHibernate.Cfg.ConfigurationSectionHandler, NHibernate"/>
<section name="syscache" type="NHibernate.Caches.SysCache.SysCacheSectionHandler,NHibernate.Caches.SysCache"/>

NHibernate specific configuration.
<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">


<property name="hibernate.connection.connection_string">Your Connection String</property>
<property name="hibernate.connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
<property name="hibernate.dialect">NHibernate.Dialect.MsSql2000Dialect</property>
<property name="hibernate.connection.driver_class">NHibernate.Driver.SqlClientDriver</property>
<property name="hibernate.connection.isolation">ReadCommitted</property>
<property name="hibernate.cache.provider_class">NHibernate.Caches.SysCache.SysCacheProvider, NHibernate.Caches.SysCache</property>
<property name="hibernate.cache.use_query_cache">true</property>

Sets up class and configuration caching for domain classes.
These cache definitions can be tweaked to change how each class is cached.
<class-cache class="Core.User, Core" region="User" usage="read-write"/>
<collection-cache collection="Core.User.Roles" region="User.Roles" usage="read-write"/>

<class-cache class="Core.Role, Core" region="Role" usage="read-write"/>



Defines Syscache specific configuration

<!-- Class cache regions -->
<cache region="User" expiration="60" priority="3"/>
<cache region="User.Roles" expiration="60" priority="3"/>
<cache region="Role" expiration="60" priority="3"/>



To explain this in a little more detail, the first section deals with telling the application about the configuration sections that will be included in the config file and what handlers to use to interpret them. This is really just standard .NET confg stuff:

<section name="hibernate-configuration" type="NHibernate.Cfg.ConfigurationSectionHandler, NHibernate"/>
<section name="syscache" type="NHibernate.Caches.SysCache.SysCacheSectionHandler,NHibernate.Caches.SysCache"/>

Then we deal with the main NHibernate configuration inside the hibernate-configuration section specified in the configsections:

<property name="hibernate.connection.connection_string">Your Connection String</property>
<property name="hibernate.connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
<property name="hibernate.dialect">NHibernate.Dialect.MsSql2000Dialect</property>
<property name="hibernate.connection.driver_class">NHibernate.Driver.SqlClientDriver</property>
<property name="hibernate.connection.isolation">ReadCommitted</property>
<property name="hibernate.cache.provider_class">NHibernate.Caches.SysCache.SysCacheProvider, NHibernate.Caches.SysCache</property>
<property name="hibernate.cache.use_query_cache">true</property>

The key things to note here are the hibernate.cache.provider_class and hibernate.cache.use_query_cache properties which are indicating that we should be using the SysCacheProvider for caching and that the query cache should be enabled.

It’s important to note that just adding the hibernate.cache.provider_class property and configuring some of your classes will cause those classes to be put into the second level class cache so that when you attempt to load them using ISession.Get or ISession.Load the the database will only be hit if the matching entity is not found in the cache.

What it will not do, however, is provide any optimisation when you attempt to load entities using an HQL or ICriteria query. Queries will still go to the database to get their results unless, of course, you add the hibernate.cache.use_query_cache property to your configuration and explicitly specify that a query should be cachable using the IQuery.SetCachable(true) method – this must be done explicitly for each query you want to be cached.

When a query is cached the entity IDs in the results of a query will be stored along with the query itself in the second level query cache so that if it’s executed again it will take the list of IDs from the cache and will then re-hydrate the entities from the class cache. Because of this the query cache is useless without the class cache.

The next section deals with configuring the how each class should be cached. This is still part of the hibernate config section:

Sets up class and configuration caching for domain classes.
These cache definitions can be tweaked to change how each class is cached.
<class-cache class="Core.User, Core" region="User" usage="read-write"/>
<collection-cache collection="Core.User.Roles" region="User.Roles" usage="read-write"/>

<class-cache class="Core.Role, Core" region="Role" usage="read-only"/>

Essentially there are two separate caching definitions to work with; the class-cache element, which deals with how a specific entity type should be cached and the collection-cache element that specifies how an entity’s dependent collection should be cached.

In this example I have elected to cache my User and Role entities as well as the Roles collection on the User entity.

Note that for each mapping you can specify a cache region, which provides additional granularity regarding how specific types should be expired from the cache and a usage that can provide some additional optimization based on whether or not the entities will ever be updated or new entities created by the application.

The final section of my example deals specifically with configuring the SysCache caching provider:

Defines Syscache specific configuration

<!-- Class cache regions -->
<cache region="User" expiration="60" priority="3"/>
<cache region="User.Roles" expiration="60" priority="3"/>
<cache region="Role" expiration="86400" priority="3"/>


Here you can define the cache regions used in the class-cache and collection-cache mappings above as well as any cache regions you decide to use in queries inside your application. Each cache region describes how long items in that region should stay in the cache before being expired so they will be reloaded again from the database and a priority that indicates which classes should be expired from memory first if the available memory is getting too low.

You can, of course, dump everything into a single cache region, but then all entities will be treated equally in the cache. In my case I want the User entities and their Roles collections to be cached for one minute as this information could be updated externally and I want to have the changes reflected in the application fairly quickly. The Role entities, however, will hardly ever change so I’ve chosen to cache them for a whole day before they should be re-obtained from the database.

Hopefully that clears things up a bit. If there are any questions or comments please feel free to post one – don’t be shy!

Feb 27 2008

Layout Change

Tag: UncategorizedSymon Rottem @ 5:46 am

Hello Reader!

You may have noticed a slight change to the layout of this blog – do not be alarmed! This change has been made for your own protection…

In fact, the main reason for the change is that I was getting very frustrated with the narrrow, fixed-width nature of the previous template I was using. I decided I had move to a layout that would not impact readers on a small screen (Nico, that’d be you!) while still allowing those with larger screens (me!) to be able to view the information without pain.

This layout is a stock template from the WordPress archive and may well be temporary pending my finding the time to roll my own.

In the meantime, sit back, relax and try not to panic. :)

Feb 23 2008

NServiceBus is Growing Up!

Tag: Open Source,SOASymon Rottem @ 4:52 am

NServiceBus is an Open Source Enterprise Service Bus framework released not so long ago by Udi Dahan and this week it got to it’s feet and wandered over to it’s new home at http://www.nservicebus.com. The site is still under construction, but it’s great to see a .NET OSS solution so recently released moving ahead so rapidly.

Over the last few months I’ve been evaluating it for use in an application and it’s certainly got a lot going for it. Configuration is not difficult and is planned to become even easier and using it is child’s play. There’s support for long running multi-message sagas, publish-subscribe semantics, pluggable transport layers, a vastly simplified and transport agnostic programming model and more – it’s definitely worth a look if you’re planning on building a scalable solution.

The only thing it’s been lacking so far has been documentation – a few months ago I submitted a patch with some code docs that might be used for generating API docs, but they’re a drop in the ocean and I really did them because I needed to read the code to really understand what was going on under the hood and marking up the methods with some code docs just seemed like a good idea to help me grok the whole thing.

What’s really needed is some guidance and tutorial style documentation and it looks like that’s beginning to get underway with the new site. Regardless, the source package comes with some great sample applications that demonstrate solutions to various common problems and since reading well written code is about the most basic method of revealing intentions the samples can work well to get you off the ground pending those additional docs.

I recently wrote a post called Do I Need Message Prioritisation? in which I discussed NServiceBus and some of the difficulties I was experiencing – I don’t think I gave it enough credit since it may not be the perfect fit for my specific problem, but I can certainly see it’s value and have plans to make heavy use of it in the future.

It’s really nice to have OSS alternatives available that don’t tie you to one specific vendor and I highly recommend NServiceBus as a software project to make friends with if you’re looking at writing scalable applications in an SOA context.

Feb 21 2008

Duplicate Madness

Tag: NHibernate,ORMSymon Rottem @ 4:08 pm

One of the things that really kills me in NHibernate is the way that when using joins there are often duplicated entities in the result set. The thing that set me off today is a thread I got involved in on the NHiberate forums where the poster was getting what appeared to be bizarre entity duplication in a list property of an object retreived using ISession.Get.

In this particular case the model was…

Person -< Order -< OrderItem

…where each Person had multiple Orders and each Order had multiple OrderItems. Each of the collections (Person.Orders and Order.OrderItems) had been mapped as bags and for performance reasons (presumably appropriate to their context) the chosen fetch strategy for the collections was set to fetch=”join”.

Everyone with me so far?

The problem here is that if you perform an ISession.Get on a Person who has 1 Order and that Order has 3 OrderItems the Person.Orders collection will contain 3 of the same Order. Why? Well this stems from the way the underlying result set returned by the database works – NHibernate formulates a query with joins in it to get all the data in one go (since that’s what we asked for).

The SQL query for ISession.Get(typeof(Person), 300) would look something like this (select clause omitted for brevity):

...FROM Person p INNER JOIN Orders o ON o.PersonId = p.Id INNER JOIN OrderItems i ON i.OrderId = o.Id WHERE p.Id =300;

(Note that this is hand rolled. And they might not be inner joins, but you get the idea.)

Using our scenario of a Person with 1 Order which has 3 OrderItems from above the result set might look like this:

p.Id p.Name o.Id o.PersonId o.Date i.Id i.OrderId i.Description
300 Symon 595 300 21/02/2008 9876 595 Fishing Rod
300 Symon 595 300 21/02/2008 9877 595 Hat
300 Symon 595 300 21/02/2008 9878 595 Boat

Now, you see the Get operation only returns one Person object but the Orders collection is populated from the result set and there are 3 rows there that contain order information so 3 order items are added to the collection.

Now it’s not quite as bad as it looks. The database operation was efficient because we didn’t have to go to the database multiple times and if you’re thinking we’re using up a chunk of extra memory with the unwanted extra Order items you’d be wrong – the list contains 3 references to the same object in memory, so there’s really only one Order there.

In this particular example the duplicates would not be there if the collections were mapped as Sets rather than Bags. This is because a Set does not allow duplicates so they are filtered out when they’re added.

A more likely situation where you’re going to see this behavior is where you use the default fetch strategy in your mapping but you execute an HQL or ICriteria query with a join:

.SELECT k FROM Cat c WHERE INNER JOIN c.Kittens k WHERE c.Colour = :ParentColour AND k.Colour = :KittenColour

When you call the IQuery.List() method this will return a list of Kitten entities, but since there can be more than one Cat that has the same Kitten in their Kittens collection (Mummy and Daddy, right?) the same Kitten may show up more than once in the result set.

This is, apparently, behavior by design. When the original Hibernate (Java version) developers were making decisions they felt that it made more sense for the returned objects to match the result set – I can’t remember the post I read it from since it was more than a year ago now, so I might not have this exactly right and I’d be happy for someone to correct me.

Again, the solution is to feed the results of the returned IList into a set which will remove the duplicates for you. There is also a feature on the ICriteria API that allows you to provide a result set transformer too – if you don’t believe me you can re-read the documentation…

Feb 16 2008

Do I Need Message Prioritisation?

Tag: SOASymon Rottem @ 11:56 am

For a while now I’ve been working on modelling a service based application that accepts messages and sends them to a set of processing services that are spread across several hosts. One of the requirements is that the application should be able to provide different service level agreements for some message originators.

Since this looked like an Service Oriented Architecture (SOA) I started looking at an Enterprise Service Bus (ESB) to standardize the model for communicating between the applications, provide publish-subscribe semantics and allow me to abstract away the underlying message transport system.

At just the right moment Udi Dahan announced that he was releasing NServiceBus, his ESB implementation for .NET, so I started playing around with it. It’s OSS and written in C# but it was pretty new at the time I started looking at it so there wasn’t much documentation available. Regardless, it looked like a good fit so I rolled up my sleeves and started rummaging around in the source and playing with it…and it still looks like a good fit.

Everything seemed to be going well with the solution design until I looked at the SLA requirement and discovered that NServiceBus doesn’t have any notion of message priority. Since my initial approach had taken me toward using MSMQ as the underlying transport and MSMQ has some basic message priority support I thought I’d be able to use it from NServiceBus, but apparently not. A possible workaround was to implement separate queues for separate priorities but since MSMQ for the versions of Windows I’m targeting doesn’t provide remote transactional read it would be necessary to implement dispatchers and the proliferation of queues was starting to look pretty daunting.

So, next stop was to ask Udi so I dropped him an email. After a reasonably lengthy thread where he essentially suggested that trying to use priority in a queuing environment is probably not a great idea for several reasons he said he’d like to think further about the problem. Imagine my surprise and pleasure when he responded with a podcast entitled “Message Priority – You Aren’t Gonna Need It” in which he dedicated 20 minutes to provide further guidance.

His response was well worth listening to and answered a lot of the questions that had been banging around in my head but, as with most of Udi’s podcasts, it raised a whole new set of questions. The net result is that NServiceBus may still work for me but I think I need to start examining other message transports to find an easier path to my solution.

Now I find myself trying to wrap my head around space-based architectures and am looking for a .NET based implementation – if anyone knows of one please post me a comment.

Feb 13 2008

Being a Better Developer

Tag: UncategorizedSymon Rottem @ 5:56 pm

The other day Ben Schierman had the following to say on his blog post about interviewing developers:

The technical responsibility we have is too great to leave in the hands of those who don’t know what they are doing and refuse to learn.

I couldn’t have said it better. As individuals and a community we have need to keep moving forward and to try be better developers all the time.

Sure, some days we don’t have the get-up-and-go to chow down on new knowledge, but if your get-up-and-go got up and went and won’t come back then maybe you need to find something you can be more passionate about.

Feb 12 2008

O/R Mapper Choices

Tag: NHibernateSymon Rottem @ 3:11 pm

I saw this article by Mario Van Damme the other day about some of the decisions you should probably take into account when choosing an Object Relational mapping tool.

Best Practices for Object/Relational Mapping and Persistence APIs

Although published in 2006 it’s a good article on some of the considerations that I didn’t take into account when choosing an ORM and probably should have. Luckily, NHibernate has worked out as a good choice overall

Mind you, when I started with NHibernate I did struggle against transparent persistence – to my mind I should have full control over when all persistence actions take place, so I did everything I could to disable transparent persistence – to my detriment. That said, I’ve learned from my mistakes and have fully embraced NHibernate’s transparent persistence and transactions and all the performance advantages it provides, however there have certainly been some projects where an persistence solution that allows me full control would have been a better choice.

Had I read the Mario’s article I might have had some idea of where I was going before I started.

Feb 10 2008

DDD and NHibernate

Tag: DDD,NHibernate,ORMSymon Rottem @ 12:35 am

I was trawling around the net today and stumbled over a series of posts (some of which are now quite old) by Ben Scheirman about developing applications using DDD with NHibernate. I don’t recall having seen this series when I was starting to get my head around both of these concepts and they would have been invaluable at the time so I thought it would be worth re-posting the links to the posts for posterity’s sake and for those developers out there who are just getting started.

The series was (after having the name refactored a couple of times) called A Journey with Domain Driven Design (and NHibernate). In the series Ben manages manages to cover a few design patterns, some TDD concepts and examples, separation of concerns, configuring NHibernate, writing mapping files and pulling it all together.

Following are the links to the articles and a quick overview of what each part covers:

  • Part 1 – Describes the reasoning behind his approach and the project structure used in the series.
  • Part 2 – Covers the application and its requirements and begins designing the domain model.
  • Part 3 – Demonstrates writing basic unit tests for the domain model.
  • Part 4 – Demonstrates more sophisticated tests that include interactions between domain entities.
  • Part 5 – Database design strategies, configuring NHibernate and starts unit testing the persistence assembly .
  • Part 6 – NHibernate class and component mappings and some tests to check they’re working as expected.
  • Part 7 – NHibernate collection and association mapping.
  • Part 8 – Some refactoring, the introduction to use of the Repository pattern and use of the Template Method design pattern.
  • Part 9 – Wrapping up the feature list and a link to download the latest version of the code.

The series is excellent and recommended reading if you are trying to get off the ground with DDD and NHibernate.

Feb 09 2008

Configuring NHibernate Caches

Tag: NHibernate,ORMSymon Rottem @ 3:05 pm

A little while ago I decided it would be prudent to start using the second level cache in NHibernate to improve performance in a couple of web applications my team and I had written. The database was taking quite a bit of punishment and performance was degrading while the data being queried was often fairly static in nature so caching seemed like the right choice.

I’d seen some samples out there where people had added the cache configuration directives in their .hbm files so off I went and started adding the <cache> element to my class and collection mappings, configured the SysCache provider in my configuration file and voila, we were off and running with the first application. Then I sat down and started thinking…

Now you are probably asking yourself the same question I was – why didn’t I sit down and think first? An excellent question – one I’ll sit down and think about. Hmmm.

The problem was that the applications in question share the same domain model assembly, which in turn share the same NHibernate mapping files but each application has it’s own caching requirements so I couldn’t reuse the new cache settings between applications. For one application we don’t intend to ever write values to a particular class so they could be mapped to the cache as read-only, while in another application the cached value for the same class should be read-write. Since we were looking at using caching for performance reasons it certainly seemed to make sense that we use the cache in the most performant way possible.

After doing a bit of extra reading of the NHibernate docs I realized that you don’t have to add you cache directives to your mapping classes; there’s a single line in the documentation I had missed:

Alternatively (preferrably?), you may specify <class-cache> and <collection-cache> elements in hibernate.cfg.xml.

I guess the lesson here is read the documentation more carefully…

So, after adding the class and collection cache mappings to our NHibernate configuration we also added some cache regions with appropriate expiry periods for each application and turned on the query cache and started up the applications again. The performance difference is quite something – the database load, which was moderately heavy, dropped significantly and the response times for the applications in question suddenly jumped. I never did do any real comparison measurements but the difference was significant enough.

I have noticed, however, that when using the SysCache provider and setting expiration that sometimes the objects aren’t expired from the cache as promptly as I would have expected. I haven’t yet investigated this issue in depth, but it’s nagging at me. I’ll report back when I have more concrete data, but if there’s anyone out there who can shed some light I’d be glad for some feedback.

Next Page »