I have given Battles For Empire a brief test drive, but these are principally written for the post-1870 era, and I'm not entirely sure whether they will work for pre-1860 games, although I've seen them used for Roman cohorts against ancient desert tribes. I may return to these if I get nowhere with the other options outlined below, as they are not a bad set of rules.
I'm looking for something that has a little bit of flexibility in that it can cover the entire period from the end of the Napoleonic Wars through to the Boer War. A great deal changed in warfare and colonial warfare during this time period, in particular the introduction of rifled muskets, breech loading rifles, and the machine gun. The rules need to be able to do justice to colonial battles, AND be scalable, in that they can cover major engagements like Isandlwana as well as the smaller actions like Rourke's Drift, where the British had less than 140 effectives.
As a bonus, the rules should also be able to handle the American Civil War, although this is very much a project for another time.
Ideally we want something that can manage the later colonial period, characterised by the arming of imperial troops with breech loading rifles in the 1870s, as well as the earlier smooth bore musket campaigns, particularly in India. A final note - the rules should be able to handle battles in which no Western armies took part, like Naushera (Sikhs versus Afghans) or Gallabat (Abyssinians versus the Mahdists).
Another criterion is that units should not be too big in terms of the figures required. A typical single formation should be no more than 8-12 figures maximum, whether it represents 100 men or 1000 men.
So what do we have? Two sets of rules appear immediately, and there is a possibility of a third.
Black Powder (Rick Priestley & Jervis Johnson, Warlord Games, 2009)
First off, we have Black Powder (Warlord Games, 2009). BP is an odd beast. Co-authored by Rick Priestley, one of the co-authors of the original Warhammer Fantasy Battle in 1983 and the creator of the original Rogue Trader rules, it is meant to be the next iteration of the Warmaster system of 10mm fantasy rules, adapted for use in the black powder era, namely 1700-1900.
This is quite a broad period to cover, and leaves the rules open to accusations of genericity, but they seem to have been well received in the five years since they were published, and are still being widely played. They include specific rules for some non-European formations, and the addition of a Zulu War supplement means that this particular campaign is well catered for with more rules adaptations and scenarios. Check out page 152 of BP, and you will see a scenario about a hypothetical action in the aftermath of the massacre of the 80th Foot at Intombi Drift. Most colonial gamers seem to be using BP these days, and Warlord have recently published a second colonial supplement, covering the Sudan campaigns (Blood on the Nile).
Field of Battle (Brent Oman, Piquet Inc, 2006)
Next, we have Field of Battle (2006), authored by Brent Oman. FoB has been very well-received since it first came out. It covers the same era as Black Powder, but uses a card-based activation system, rather than BP's dice-based system. FoB is a simplified version of Piquet, which is a more generic set of rules, supported by period specific supplements. FoB was a response to criticisms of the impetus system employed by Piquet, and is considered the more playable and accessible of the two. FoB should be able to manage most actions from the colonial period, although its Sikh Wars army list is fairly sparse, and there is nothing on the Indian Mutiny or the First Afghan War, despite there being three pages of Napoleonic army lists. These Indian lists will need to be created from scratch. Also, I'm in some doubt as to whether FoB can manage smaller actions, i.e. with multiple units forming one company. One intriguing aspect of FoB however is that the card activation deck can be tailored to reflect specific army states or unique factors that really only applied to one engagement.
Fields of Honor (Shane Lacey Hensley, Pinnacle Entertainment Group, 1994)
Role players will recognise Shane Hensley as the creator of the Deadlands RPG and Savage Worlds, but before all that he wrote Fields of Honour (which I will now spell in the English rather than American manner). Now out of print, FoH sets out to cover the era from 1815 to about 1900. It does everything I want, including simulating both conventional warfare, like the American Civil War, and colonial campaigns. And yes, there are army lists for the First Afghan War, and Sikh Wars, and the Indian Mutiny. There is even a scenario for Isandlwana, although I don't think much of it. But this has it all, including the ability to scale up from about one squad to the base, to an entire regiment. This is excellent, because some of the battles of the Sikh Wars were pretty big as colonial wars went (an estimated 10,000 Sikh and allied troops fought at Mudki in 1845, and there were over 30,000 at Ferozeshah, including irregulars and feudal levies). FoH is probably not as specific to the period as, say, Chris Ferree and Patrick Wilson's John Company (Virtual Armchair General, 2005), but JC would be useless for the Zulu War.
We'll see how far we get with this project. Ultimately, we could be looking at up to six play tests here, and I'm doubtful whether I will get the time this year. Further progress will be duly reported in these august pages.