Since we submitted the Beeline app to the Store, we have been working hard on v2.0 of the web UI. We just shipped the new, improved version and you can find it at www.beelineapp.com.
All the functionality that is available in the iPhone app is now accessible through the browser, making it even easier to manage your mobile team.
We would love to hear from you if you have any questions, comments or suggestions:
The Beeline App has been approved and is available on the app store. Beeline tackles the problem of communicating and organizing a small mobile workforce. You can allocate jobs, map their location and provide a simple, realtime line of communication inside your team.
Check it out, we would love to hear what you think.
As with any server installation, a massive feature (often overlooked) is the need for good server side logging. That is, the ability to work out what is being used, by whom and for how much. Even more important is to be able to find out what went wrong, in case of any (ahem) bugs.
We already have procedures and systems in place for our Windows installations, but as this is the first time we’ve deployed a massively available server on Linux (as part of Goldfish). As such, we needed to make sure we can backtrack any problems correctly and efficiently.
As a result, I thought I’d dedicate a post to Linux logging facilities, in their most simple state: Enter syslog.
Syslog is the Linux System Logger and is exceedingly easy to use from a C++ perspective: open the log (optional), write to the log and close the log when the application exits (which in our case, is never).
Using C++ syslog:
#include <syslog.h>
openlog(“goldfish”, LOG_PID|LOG_CONS|LOG_NDELAY, LOG_LOCAL0);
…
syslog(LOG_WARNING, “Down to %d free bytes of RAM”, ramLeft);
Notice how in openlog() we specified a facility code of LOCAL0? This becomes important later when we configure a custom log.
On Ubuntu systems (which is what we use for development), you can easily see the log messages in the Log File Viewer (System -> Administration -> Log File Viewer), updating in realtime.
However, to generate a ‘goldfish.log’, a small configuration change has to be made.
In /etc/rsysconf.d/ create a file called ‘goldfish.conf’ and add the following text:
local0.* -/var/log/goldfish.log
This means that anything arriving at the system log using facility code LOCAL0, it should get added to the goldfish.log. Since we know that nothing else is logging toLOCAL0 this means that only goldfish messages arrive there.
Note that we could have chosen any number from 0 to 7, eg LOCAL7, in case of a conflict with some other application.
Note also that the ‘-’ near /var/log/goldfish.log indicates to enable write-behind caching of the log, meaning that the log is not immediately written out to, just periodically. This reduces the hard drive thrashing itself, while it tries to save the logs immediately when a new message is received.
So, you may ask, “Ok, so now we have a log file being generated,” won’t that just fill up the hard drive and then fail in a cataclysmic manner? Not so, there is generally a process called ‘logrotate’ which knows exactly what to do, without any fuss.
You may also ask, “So, how do I watch the log”? Simply use the ‘tail’ command:
tail -f /var/log/goldfish.log
One of the most frustrating things I have encountered whilst developing C++/CLI controls is getting a simple ToolboxBitmap to work correctly! Yes, it’s ironic I know, to find that one of the first things (or the very last thing) that you need to do as a control developer is to get your graphic artist to design an icon for your control and then allow it to be displayed in the Visual Studio Toolbox like a normal control (like a Button, ComboBox or DataGridView, for example).
If we were developing in VB.NET or C#, we find that this is a simple task. Unfortunately, in C++/CLI the process is awkward and there is nothing anywhere that describes how it is done. Several developers on the web have also encountered this issue with no appropriate responses.
In a nutshell, Microsoft suggests the form:
[ToolboxBitmap(MyControl::typeid, "MyControl.bmp")]
public ref class MyControl : public Control {
...
};
…and specifies that the “BuildAction property set to EmbeddedResource“. This is of course a bogus comment for a C++ project which has no such property of a bitmap.
C++ developers would then naturally starting thinking of a Resource File (.rc) and include the image in that.
After struggling with that route, a C++ developer might also try using an Assembly Resource File (.resx). Again, developers will quickly fall flat on their face with a profound lack of progress.
So without further ado, here’s what you need to do to get your ToolboxBitmapAttribute working correctly in C++/CLI.
Embedding an Image in C++/CLI
Assuming you have a control in a nested namespace :
namespace MyCompany { namespace MyControls {
public ref class MyControl : public Control
{
...
};
}} // namepace MyCompany::MyControls
You have two options to get your image working correctly:
Using a nested Namespace
Your code should look like the following:
namespace MyCompany { namespace MyControls {
[ToolboxBitmap(MyControl::typeid, "MyCompany.MyControls.MyControl.bmp")]
public ref class MyControl : public Control
{
...
};
}} // namepace MyCompany::MyControls
Using the Global Namespace
This was an idea I found from a C# developer who recommeded using a Global Namespace idea. This means that you can simply name your bitmaps logically (eg “MyControl.bmp”). See here for the article.
Your code should look like the following:
ref class resloader { };
namespace MyCompany { namespace MyControls {
[ToolboxBitmap(resloader::typeid, "MyControl.bmp")]
public ref class MyControl : public Control
{
...
};
}} // namepace MyCompany::MyControls
A Quick Note
Happy Coding!
References
Thanks go to the following to the developers who gave me the hints needed to find the solution:
Maybe this article should have been called, “.NET Controls - Things that trip you up” or “.NET Controls - Making your life easier”. Regardless, this is a small compendium of issues that recently arose whilst I was writing an Interop layer over some Win32 Windows and COM Objects using C++/CLI.
Essentially, the mini-project was to convert some Win32 windows that we use regularly for inhouse purposes (eg a super-fast grid that can handle oodles of updates/sec). Some of these Win32 windows also have COM counterparts and these needed to be made more friendly.
In Win32 you create a window by calling CreateWindowEx() specifying the Window that you want to create. There are also many articles that can be found that use P/Invoke to do just that.
The code-happy fellows at Microsoft have also recently released WPF as part of the .NET Framework v3.0. In that they define a class HwndHostthat apparently provides a similar result. If you are interested in that, you can check out an intro here.
Our clients however, are reluctant to apply the most recent update of the .NET framework, meaning that we needed to be .NET 2.0 compatible (even though v3.0 is v2.0 with some extra functionality) . Thankfully, using C++/CLI one doesn’t need to worry about P/Invoke at all and since WPF is actually an extension onto .NET 2.0 we should be able to re-invent what WPF is doing by ourselves.
Getting started with C++/CLI
For those not yet ready to unleash the beast of writing for .NET in C++ here’s a quick primer:
That should be enough words for the time being. Now lets look at writing a .NET control that uses an existing Win32 control.
Step1. UserControl or not UserControl
You may initially think to start Visual Studio and create yourself aSystem.Windows.Forms.UserControl, because that’s what you traditionally do when creating a .NET control. However, what we are actually doing is creating an Interop between Win32 and .NET architecture. So, we only need to use a System.Windows.Forms.Control rather than aSystem.Windows.Forms.UserControl. Have a look at the Class Browser for both of the classes and you’ll see what I mean: most of the meat that goes into creating a UserControl is actually presented in the Control class.
Step 2. System.Windows.Forms.Control initialises in a manner similar to a COM component (ActiveX object).
That is, once your control’s constructor has been called, you will get a slew of other things happening to your control before it actually gets created with Control::OnCreateControl(). Let me elaborate:
This of course is blatantly obvious if you look at what happens to a control when it is added to a Form. If you create a simple Windows Forms project and drop a button onto the form you will see the following inInitializeComponent() in Form1.Designer.cs:
this.button1 = new System.Windows.Forms.Button();
this.button1.Location = new System.Drawing.Point(98, 263);
this.button1.Size = new System.Drawing.Size(75, 23);
…
Knowing that InitializeComponent() has been called from your Form1’s constructor, you can see how OnCreateControl() hasn’t (and can’t) be called until later in the processing loop.
Fixing Step 2.
Simple. Simply create your Win32 control in your constructor.
This of course raises the immediate issue of ‘How do I get the Form’s parent window handle’. Again, the Form hasn’t actually been created yet either (because when the Control’s constructor is called, we are still in the Form’s constructor), meaning that trying to get the Form’s window handle is pointless.
The solution then is to do the following:
Use C++ Interop and call GetDesktopWindow() (a Win32 function that returns the HWND of the Desktop) to return a valid window handle. (We do this rather than using NULL since many Win32 controls do not like being instantiated without a valid parent). Essentially, you’ll have a call that looks like:
m_hWnd = CreateWindowEx(exStyle, “MyWin32Control”, …, GetDesktopWindow(), …);
Just make sure that you don’t specify WS_VISIBLE at this point, since it (potentially) can cause a flicker on your desktop.
“Cool!” I hear you say, “but that’s in the wrong place!”. Yes, it is, but we will fix it later.
Now, override OnCreateControl() and and add the re-parenting code there:
virtual void OnCreateControl(void) override
{
Control::OnCreateControl();
HWND hWndNewParent = (HWND)(Handle.ToPointer());
::SetParent(m_hWnd, hWndNewParent);
}
Easy, no?
In my next articles, I will touch upon the following topics:
Microsoft Excel is undoubtedly an extremely powerful, compelling application and has often been described as a killer app for the Windows platform. It is no wonder as the spreadsheet paradigm is certainly the original killer app, and built on top of this foundation is a powerful array of functionality to extend and customize the Excel calculation engine. However, with great power comes great responsibility. An unfortunate, but common, characteristic of Excel’s vast customer base is complex proprietary systems, built up and modified over many years, that are a constant source of anxiety for those unfortunate souls who are tasked with their maintenance.
So what is it about Excel that nurtures these sometimes horrific, ethereal and interdependent creations? What is it that compromises the robustness and recoverability of these solutions?
Excel is often used to consume information, to model complex systems with that information, and then to publish that information to some kind of external repository, where it can be viewed or consumed by other systems.
To import data into Excel we could use:
DDE links
Long since deprecated, nevertheless still supported and often used to stream realtime updates into Excel. The DDE technology is based on data attached to Windows messages and passed between processes, with a clearly defined protocol for requests and responses. Excel acts as a DDEclient when the appropriate links are inserted into Excel, the links themselves rely on the DDE server for flow control and do not support parameter references inside the sheet. Microsoft describes DDE as “not designed for getting real-time data into Excel in a robust and high-performance way” yet it is widely used and DDE servers for proprietry data sources are very common.
RTD links
RTD was developed by Microsoft as a replacement for DDE, a solution that would “overcome the performance and robustness issues with DDE”. This technology is based on an RTD Server object (a proprietry COM DLL) instantiated in process by Excel. The RTD Server object can notify Excel when it has updates available and Excel applies flow control and aggregation on those updates according to global settings. RTD links in Excel support parameter references and can be wrapped with VBA functions to create a consistent namespace for an Excel addin.
Custom Solutions
Then we have proprietary data source solutions based on a spectacular variety of ingenious custom solutions that take advantage of VBA, automation etc…
Cold Excel Spaghetti
Already, we have a fair degree of complexity getting data into the Excel environment. This pales in comparison with the casually created, and impossibly complex, networks of dependant Excel cells, chains of references, inline formulas that concatenate, manipulate, obfuscate data spread across sheets and workbooks far and wide. Anyone who has tried to analyse a spreadsheet created many years ago, by that bearded guy who had the Star Wars theme chess set on his desk and left the company to write spaceflight simulation software for NASA, will know what I mean.
This complexity is again compounded by the limitless variations for exporting or publishing the results of an Excel based system. This ranges from text files, financial services publishing systems, databases or indeed automating Excel to generate additional spreadsheets.
These kind of spreadsheet systems can have any number of performance, reliability and recoverability problems, and often require constant monitoring and maintenance to ensure continuing service for organization critical systems.
Performance
The ongoing increase in market data update rates is a well known phenomenon, and it’s implications for Excel based applications can be dire. Excel applications built without due consideration for update rate flow control, and the implications of the Excel dependency engine, will not scale, and therefore fail at critical moments. Realtime data sources for Excel applications (DDE, RTD and custom) must be aware of throttling for update rates for these applications to continue to function as traffic increases. Systems that snap data from external sources based on Excel dependencies must be aware of the implicit volatility of that dependancy engine. These type of systems need to be carefully driven by other throttled polling mechanisms (timed RTD updates), rather than the dependancy engine.
Reliability and Recoverability
The variety of external data sources that is available to applications built in the Excel environment necessarily results in a great variation in the quality of service these data sources provide. That quality of service is transfered directly into the Excel application. Data sources that are unaware of performance implications can drive Excel to its process bound knees. Workbook dependencies on network drives will predictably fail at inopportune moments and bring critical systems grinding to a halt. Excel flexibility means that developers in the Excel environment need to exhibit constant vigilance when tempted by accessing data resources directly, without the protection of fault tolerance and recoverability systems.
Absolute power corrupts absolutely?
I hope not. It is undeniable that Excel is a powerful and vital tool for any business. I certainly don’t suggest that the flexibility that Excel delivers should be curtailed in any way. Instead, I think that the challenge, as a programmer, is to build frameworks for these Excel-based applications that facilitate creative development, whilst protecting the applications from Excel’s inherent pitfalls.
ASP .NET 2.0 comes with a handy feature called ASP pre-compilation. It is described in detail on a number of pages:
http://www.odetocode.com/Articles/417.aspx
http://msdn2.microsoft.com/en-us/library/ms228015.aspx
There is even an add-on for Visual Studio 2005 that can do the pre-compilation for Web Application Projects:
http://msdn2.microsoft.com/en-us/library/ms228015.aspx
Pre-compilation
This is all well and good but what if you have a non-trivial site with multiple Visual Studio projects and multiple virtual directories? The good news is that it is possible to pre-compile any complex project for deployment, the bad news is that the of doing so is not as straight forward and as flexible as one hopes it is.
The scenario described in this blog probably applies if you have projects that are converted from Visual Studio 2003. I am going to use a sample project with three projects - Main, FrameworkControls and MyControls. Main project contains pages and controls relevant to this application, Controls contains some controls and code that you’ve written and FrameworkControls is a project that is shared with another team and contains some web user controls and some code. Let’s also assume that Main uses controls from MyControls which in turn uses controls from FrameworkControls.
Below is a map of the web site that displays how the virtual directories are mapped.
Project directory………..Virtual directory
Main……………/
MyControls
+www…………controls
+src
FrameworkControls
+www…………framework
+src
Let’s now try to pre-compile it.
aspnet_compiler -p c:\Precompilation\Main c:\Precompilation\deployed
The above obviously wouldn’t work as the compiler doesn’t know how this physical folder is mapped to a virtual folder in the IIS. In fact it will tell you to specify either -m(metabse path) or -v(virtual path) with the -p command. Let’s try to specify the virtual path:
aspnet_compiler -f -p c:\Precompilation\Main -v / c:\Precompilation\deployed
This fails with this rather interesting error:
/Default.aspx(2): error ASPPARSE: The file ‘/controls/FramedButtonLabel.ascx’ does not exist.
/Default.aspx(15): error ASPPARSE: Unknown server tag ‘ctrls:ButtonLabel’.
The ASP compiler now knows where the root folder is mounted, however since the physical folder is specified, it will only look within that folder and hence not find other virtual folders. This is fixed by running the following command:
aspnet_compiler -f -v / c:\Precompilation\deployed
If there are no errors in the pages the site will compile. Note that the output of the pre-compilation is the contents of the Main folder with the folder controls and framework folders inside it, in other words, as IIS “sees” it. Alternatively you can specify the metabase path:
aspnet_compiler -f -m /LM/W3SVC/1/Root/ c:\Precompilation\deployed
The metabase path might come in handy when using the web deployment projects.
Unfortunately with this site layout it’s impossible to compile individual projects. So you can’t compile controls or framework because they don’t have the bin folder. So any assemblies that those two projects reference will not be found.
aspnet_compiler -f -v /controls c:\Precompilation\deployed
You will see this kind of error executing the above command:
/controls/FramedButtonLabel.ascx(1): error ASPPARSE: Could not load type ‘MyControls.www.FramedButtonLabel’.
/controls/FramedButtonLabel.ascx(2): error ASPPARSE: The file ‘/controls/framework/ButtonLabel.ascx’ does not exist.
/controls/FramedButtonLabel.ascx(5): error ASPPARSE: Unknown server tag ‘fm:ButtonLabel’.
Excluding projects from pre-compilation
A neat little trick that can be done (although the usefulness of it is near zero) is an ability to exclude projects from pre-compilation. You do that by creating an application on the virtual directory you want to exclude(under the “Home Directory” tab in the virtual directory’s properties). So if we create an application for, say, controls directory we will exclude it from pre-compilation. Running the pre-compilation again for the root folder yields the contents of the Main folder with only framework directory inside it.
One of the fun aspects of life here at Cognethos is the number of different languages and environments you can find yourself working in. I’ve had projects in C++, C#, Java, HTML+Javascript, JSP, ASP.NET. Sometimes it feels like each month it is a new language and a new IDE.
Lately, I’ve been working on Craig’s port of the chart to Flash and ActionScript. This has given me a chance to compare the various approaches we’ve taken to making web content update dynamically.
In the products I’ve worked on I’ve seen four approaches so far:
‘Pure’ JavaScript
The ‘pure’ javascript approach refers to using AJAX and Comet style javascript to update the pages DOM. I’ve put the inverted commas around pure because, as anyone who tried to do cross browser javascript knows all too well, javascript is rarely pure.
This approach is hampered by needing to code around browser quirks but also by the ‘hand-built’ nature of doing large amounts of programming in javascript. There is no compile-time. You only know you have a bug or a typo in your code if you happen to manage to get your code to pass through that particular path of execution and the bug occurs in a nice enough fashion for you to notice it easily. A simple typo in a variable name can have far reaching consequences. Since javascript variables don’t need to be declared the misspelt variable will have an undefined value. If you’re lucky the bug will manifest itself in the next line or two. If you’re not so lucky it will get passed into another function and show up as a bug somewhere far from the original error. I find that with programming in javascript it is a good idea not to write too much code at once. You write a bit, you test a bit, you write a bit again. If you go off and spend half and hour writing a long screed of script and then come back and expect it to work flawlessly first go you’re going to be in for a very rude shock.
But the advantages of the approach are that it is light-weight and very portable. There are no plug-ins, just a few .html and .js files. Updating the DOM is easy. I find this approach is the best for updating text and tables, but it falls down when it come to charts and graphics. The javascript approaches for graphics aren’t standardised enough (SVG vs VML), and although some people have done great work on producing libraries to cover over the incompatibilities, when it comes to the requirements we meet for financial charting the pure javascript approach doesn’t measure up.
ActionScript
Enter ActionScript. ActionScript is like a typed Javascript. It has a compiler, so the fear of lurking bugs from typos is eliminated.
Since it is the language behind Flash player the graphics capabilities are as good as you’d expect. Have a look at the Cognethos demos for our implementation of a Flash chart.
The ActionScript language has types and the compiler will do type checking on assignments and parameter passing. This makes the code a lot neater.
The code gets compiled down into a Shockwave (swf) file. It seems to be optimised for size to make the download to the client quicker.
However, since we use javascript for our lightweight streaming data we need to be able to push data from javascript into Flash. Adobe provide a bridge that enables that.
Also since this is Flash, we need the Flash plugin, a swf file and an <object> or <embed> tag in our html. From javascript, you cannot begin to access the objects in the swf file until it has finished loading.
All in all, ActiveScript is a good environment for what we want to do. The Flex IDE is an eclipse clone, and it is (in my opinion) a bit clunky, but it is bearable.
Java applets
Which brings me to the next alternative. Why not just use a Java applet? The technology is very similar. You need to compile your java into a jar file, and use an <object> or <embed> tag to load it into your html. The jar file replaces the swf file, and both technologys use the same html tags.
Java is a much more mature language, and the development tools are better too. I use Netbeans for my java work, and find it a lot easier to work with than Eclipse.
The drawback for using Java are the graphics APIs. Java is a much more general language, and you need to do a lot more work when coding the animation, transparency and other effects.
Java is also a much more memory hungry beast. Again, I think because it has been around longer, and is now expected to do more things, the ‘bundle’ or core code that is running when you start your applet is larger than for Flash.
Another downside to both Flash and Java is the VM stability. Both plugins have been around for a while, and you would have thought the bugs would have been ironed out, but you still get things breaking from time to time when the vendor does a new release. We’ve had this happen to us a number of times, and the end result is that you have to do an emergency fix because 1.5.0_07 broke this or version 9 broke that. I’ll write another post on that one shortly.
Java driving Javascript
This is another good technique for doing dynamic updates ito HTML. The trick is to have a ‘hidden’ or 1 pixel x 1 pixel applet that handles the connection back to the host and drives the updates through to the HTML DOM.
This can be good for implementing sophisticated retry or timeout mechanisms. Its a lot easier to write reconnect logic in Java, where you can have a dedicated thread that can loop and retry, than it is in Javascript where everything has to be in a callback, or a timer or in the UI thread. It is also good for multiplexing many updates to the page to a single update from the data feed.
Conclusions
I like to use ‘Pure’ javascript as a first try. If that is good enough for what you want to do, why go further? If the application requires ‘pretty’ graphics go with Flash. If the application requires sophisticated networking go with Java and Javascript.
For the past few years, Cognethos has been using an application framework we call Loki to build most of our desktop applications. Traditionally these applications would have been built using a source code framework such as MFC or ATL - which are fine - but we had a few additional requirements that we wanted to be able to use in all future applications. For this reason we developed Loki which is a binary application framework. In other words, Loki is not a source code library for building applications. Instead it’s a container application that can be extended using a set of plugins.
At present we have three main applications that have been built using the Loki framework - Mercury, Atlas Data Capture and our Scipt Editor. As the screen shots below show, although these are all functionally very different applications - however they all have a consistent look and feel thanks to Loki.
By itself, Loki appears as pretty much an empty application with little more than a main window. Its through its plugin extensibility that the power of this framework becomes apparent.
Features of the current version of Loki include:
As well as these built-in features, we also have a set of services that provide common functionality that can be re-used in all the applications built using Loki.
Using Loki framework lets us concentrate on building the main functionality of an application, leaving Loki to provide the many other details that a high end desktop application should have.
Developing software for a mobile phone is supposed to be a snap, especially when you use the Java2 Micro Edition (j2me). However, just a quick perusal at the sun or nokia development sites, you’ll quickly see that simple “one size fits all” solutions simply don’t exist. In fact, there is a well-coined phrase: “Write once, test many”, when it comes to mobile development. Even Google Maps for mobile devices (www.google.com/gmm) show that it has had to generate several flavours of the app to satisfy the differing manufacturers. Then when you add in the telcos themselves, each with their own set of enabling/disabling mechanisms designed to restrict users to their particular mobile plan, you quickly find that any mobile solution can be quite messy indeed.
What makes the problem even worse is some of the ‘helpful’ comments that one can find on the internet that relate to a particular problem: they never take into account all of the different flavours of manufacturers and telcos (as if they could). For me, the bane of my existance is the HttpConnection object. It is supposed to be a simple way for a j2me application to retrieve web pages and/or post up some results. Not so.
Each implementation appears to be sublty different, which has profound effects on the way a telco will then interact with the headers that are sent with the Http request.
From a developer’s standpoint, I can simply add new headers into the Http request simply by calling:
connection.setRequestProperty()
Unfortunately, by doing so this can inadvertantly add a duplicate header. Need I say that some telcos do not cope with duplicate headers well at all? To compound the problem, there is no way for me to verify that I have added a duplicate entry. The best one can do is to attempt to make a connection and then see what happens when j2me fires some java exception (such as IOException). To compound the problem, ASP.NET 2.0 has a set of filters that limit the type of html content returned. I am stymied everywere!!!
In short, if I specify:
connection.setRequestProperty(”Host”, “www.mysite.com”)
Some phones will add a duplicate header, some phones won’t. Some telcos are OK with having duplicate headers, some telcos are not. Failure to specify this may result in the request not actually making it to the server, especially if the server is a virtual server (ie one physical server that handles multiple dot coms on the same machine). Damned if i do, damned if I don’t.
If I specify:
connection.setRequestProperty(”User-Agent”, “Profile/MIDP-1.0 Configuration/CLDC-1.0″)
Again, I run into the dreaded duplicate header issue. By failing to specify a User-Agent (and/or a ‘Accepts’ header) string, some websites (eg those running ASP.NET) will return incorrectly formatted xHTML (and return the old ghastly WML format instead).
As such, it appears that I will be forced to create a suite of ‘tests’ that will run on a mobile device, each testing different Http header sequences, all in the vain attempt to connect to a simple website.
More will come later.
Note that for a CogStream solution we have found that it is actually best to use a raw TCP socket connection (by using SocketConnection). Again, there are also problems with this approach, namely that rather when we use a SocketConnection and mimic the complete HTTP request headers, we can’t use Port 80. This is because attempting to open a direct connection to some server on Port80 will bypass a telco’s and the phone’s security model. So, CogStream for a mobile phone needs to run on an alternate port altogether.
