November 28, 2013

What is ProcessMessages?

Sometimes, when you browse some application source code, you find loops calling Processmessages. What’s this? Well, it is a “Message pump” which retrieves and process messages from Windows message queues. More exactly from the current thread message queues.

Quick note: At the end of this article, you'll find why you should NOT use ProcessMessages. Before that, let's understand the concept.

The code you see is frequently like this:

FSomeFlag := TRUE;
while FSomeFlag do begin
    Application.ProcessMessages;
    Sleep(100);
end;

When looking at that code, you may think it will loop forever. Actually it will not! The loop variable “FSomeFlag” is for sure set to false in an event handler.

Application.ProcessMessages is implemented in Delphi runtime (Forms unit) as a loop checking Windows message queue for the current thread. The loop removes any message from the queue and dispatch processing, until the queue is emptied.

In the message queue, you have messages from the user interface (for example mouse move, button click, menu actions and much more) and from many activities occurring in the system such as inserting a removable disk, user wanting shutting down the computer, data received from a serial port, data packet received on the network, battery running out of charge and much more.

ProcessMessages will fetch all those events and process them. Translated to more common Delphi speaking, your event handlers will be called from ProcessMessages.

Try this:
Create a new VCL forms application, drop two buttons on the form and a memo. Declare a private Boolean member variable named “FSomeFlag”. Then write and run this code:

procedure TForm1.Button1Click(Sender: TObject);
begin
    FSomeFlag := TRUE;
    Memo1.Lines.Add('First message: Start looping, click on Button2');
    while FSomeFlag do begin
        Application.ProcessMessages;
        Sleep(100);
    end;
    Memo1.Lines.Add('Second message: Loop done');
end;

procedure TForm1.Button2Click(Sender: TObject);
begin
    Memo1.Lines.Add('You clicked on Button2');
    FSomeFlag := FALSE;
end;

When running, click on Button1. This will execute Button1Click event handler which contains the loop. It will display the first message and then loop until FSomeFlag becomes FALSE.
After clicking on button1, then click on button2. This will execute Button2Click event handler which sets FSomeFlag to FALSE which in turn will break the loop still running in Button1Click event handler. You’ll see the message display by Button2Click event handler, followed by the second message in Button1Click event handler and you know the loop has ended.

This code has some surprising results. For example, click twice on Button1 before clicking on Button2. You will see the first message shown for each click on Button1. Then the message from Button2 when you click on it. And finally twice the second message from Button1Click event handler. What you see is the Button1Click event handler reentered. This is a king of recursive call. ProcessMessages will trigger all events, including new event of the same kind already executing.

Having you event handler reentered may have adverse effect on your code. You should probably avoid it by rewriting Button1Click like this:

procedure TForm1.Button1Click(Sender: TObject);
begin
    if FSomeFlag then begin
        Memo1.Lines.Add('You already clicked Button1');
        Exit;
    end;

    FSomeFlag := TRUE;
    Memo1.Lines.Add('First message: Start looping, click on Button2');
    while FSomeFlag do begin
        Application.ProcessMessages;
        Sleep(100);
    end;
    Memo1.Lines.Add('Second message: Loop done');
end;

Checking FSomeFlag before the loop let us know that the loop is already looping and just exit instead of beginning a new loop.

There are more surprises: once you clicked on Button1, before clicking Button2, try to stop your application by clicking the close window button. It won’t stop!

The application doesn’t stop because VCL is written in such a way that an application is terminated when the main window closes. And that window cannot close while an event handler is still executing. Which event handler is execution? It is Button1Click which is still looping waiting for FSomeFlag to become FALSE. Click on Button2 and the loop will break, letting the window close and the application terminate.

It is likely that this behavior is unwanted. Easy to change. Rewrite Button1Click event handler like this:

procedure TForm1.Button1Click(Sender: TObject);
begin
    if FSomeFlag then begin
        Memo1.Lines.Add('You already clicked Button1');
        Exit;
    end;

    FSomeFlag := TRUE;
    Memo1.Lines.Add('First message: Start looping, click on Button2');
    while FSomeFlag do begin
        Application.ProcessMessages;
        if Application.Terminated then
            break;
        Sleep(100);
    end;
    Memo1.Lines.Add('Second message: Loop done');
end;

The check for Application.Terminated does exactly what it name implies. Actually “Terminated” is a flag which is set when you click on the form’s close button. This is exactly the same mechanism as the one we implemented with FSomeFlag.

Finally, there is one more glitch: what happens if the user never clicks on Button2? Answer: the application loops forever. This is probably unwanted behavior (Remember that is a real application the flag FSomeFlag is set to FALSE no when a user clicks on a button but when some condition is met, for example when a removable media is inserted).

To fix this behavior, we may add a timeout. There are several possibilities to implement it. You may get the time within the loop and break if too much time elapsed. Or you may use a TTimer for that purpose. The code looks simpler with a timer but is not really visible so I prefer to get the time within the loop like this:

procedure TForm1.Button1Click(Sender: TObject);
var
    Timeout : TDateTime;
begin
    if FSomeFlag then begin
        Memo1.Lines.Add('You already clicked Button1');
        Exit;
    end;

    FSomeFlag := TRUE;
    Memo1.Lines.Add('First message: Start looping, click on Button2');
    Timeout := Now + OneSecond * 10;
    while FSomeFlag do begin
        Application.ProcessMessages;
        if Application.Terminated then
            break;
        if Now > Timeout then begin
            Memo1.Lines.Add('Timeout');
            break;
        end;
        Sleep(100);
    end;
    Memo1.Lines.Add('Second message: Loop done');
end;

I have not explained yet why there is a Sleep(100) within the loop. If you remove it, everything seems to work the same way. Really? Not! use the task manager to see how much CPU is used by your application. Since it does almost nothing, task manager shows 0% of CPU used. If you click on Button1, you still see 0%.

Now remove Sleep(100) and test again. Task manager will show your application consuming a lot of CPU just for doing nothing sensible. Why is this?

The loop in Button1Click event will run at the maximum speed when there is no Sleep call. Depending on your CPU (Number of cores), this means the loop will consume almost all CPU time available using a single core. The loop will loop thousand or million times per second depending on your hardware.

When there is a call to Sleep, it will instruct the operating system to make the thread sleep for the remaining time slice and for approximately the time in millisecond. Using Sleep(100) will make your loop not run more than 10 times per second which will no use to much CPU.


Better solution?

I explained how ProcessMessages works and how to use it. Now it’s time to ask the good question: Should I really use ProcessMessages?

Most of ProcessMessages calls should and can be avoided. They should be avoided because there is a waste of CPU which slows down your application and the computer in general. They can be avoided by redesigning your application using event driven asynchronous pattern.

Most use of ProcessMessages are used when the developer has not correctly understood or is not even aware of the event driven asynchronous pattern. Such developer has a problem with asynchronous operation where your calls merely start an operation and give control back immediately while the operation itself is executed in the background and triggers an event when done.

In the simple case I used above, it is very easy to change the code so that no wait loop calling ProcessMessages is required. Let’s assume that some processing has to be done when the user has clicked Button3 and Button4 in any order.

The idea is to use a flag per event (Button click here) that must have occurred before the processing can take place. For each event, check all flag to see if processing can take place.

Here is the code:

procedure TForm1.Button3Click(Sender: TObject);
begin
   if FFlag3 then
       Memo1.Lines.Add('You already clicked Button3')
   else begin
       Memo1.Lines.Add('You clicked Button3');
       FFlag3 := TRUE;
   end;
   TryToExecute;
end;

procedure TForm1.Button4Click(Sender: TObject);
begin
   if FFlag4 then
       Memo1.Lines.Add('You already clicked Button4')
   else begin
       Memo1.Lines.Add('You clicked Button4');
       FFlag4 := TRUE;
   end;
   TryToExecute;
end;

procedure TForm1.TryToExecute;
begin
    // Check if both buttons have been clicked
    if (not FFlag3) or (not FFlag4) then
        Exit;
    // Both have been clicked
    Memo1.Lines.Add('Processing done');
    // Reset the flags so that the user can start again
    FFlag3 := FALSE;
    FFlag4 := FALSE;
end;

There can be cases where order of event is important or there are a large number of conditions. In those cases, the whole application should make use of a finite state machine. This will probably be the subject of a future article. For now, have a look at what Wikipedia says about it: http://en.wikipedia.org/wiki/Finite-state_machine

Update 16/12/2013: I wrote another article about how to make PostMessage multi-platform for Android and Windows: http://francois-piette.blogspot.be/2013/12/firemonkey-android-windows-and.html


Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be
This article is available from http://francois-piette.blogspot.be

November 24, 2013

OOP design and event driven programming

In a previous article, one of the 3 main design advices was “Use events to free your classes from things that can change”. In this article, I would like to talk a little bit more about it.

Following good OOP practice, you develop by encapsulating almost everything into specialized objects. This makes all the code and all data related to a given subject grouped into a single entity named object.

This encapsulation is quite easy. You first this about what the object is or made of, and what operation can be done on the object or with the object. What the object is or is made of becomes properties. Operations become methods.

Let’s see this in action with a very simple and basic example derived from one of my real world applications. You can skip the following 3 paragraphs if you are not interested by the actual application and only want to read software design part of this article.

First, a little background: As you may know I’m working for a company building – among other things – automated digital radiography (DR) system. A DR system is made of an X ray source, an X ray detector, a diaphragm, a manipulator, a control computer, an image processing system and a database system. It is automated because it does X ray inspection without human interaction. It takes hundreds of radiography fully unattended.

You can see a picture here. OK, when you don’t know what it is, it could be challenging to understand what you see. Actually in the background in orange you see a robotic arm. It holds a fork which support the X ray detector (White rectangular box on the left), the diaphragm (Yellow on the right) and the X ray source (just behind the diaphragm). On the foreground, you see the part being inspected (Component of the low pressure compressor of an aircraft engine). This component is secured on a rotating table you don’t see on the picture.

What my Delphi software does is drive the robot, the rotating table, the X ray source, the diaphragm, and the detector in coordinated movement to take X ray picture of all welds. Pictures are sent to an image processing system for examination and then stored in a database for later search and retrieve.

For such a complex system, it is very important to have a good software design. If you don’t, the application will be horrible to maintain and would probably quickly become unreliable.

Objects are defined at all levels in the application. High level objects making use of low level objects. Each object is specialized for his own purpose. If you don’t carefully think about the purpose, you’ll end up with a single huge object doing everything; or you get a myriad of object doing almost nothing each one. There is no rule to fix the boundaries. Your only guide I can give you is to always think about what the main purpose of the object is and concentrate on it. Everything that doesn’t fit the real purpose must be moved to another object with his own purpose.

Back to our real world example: Both the robot and the rotating table make use of serial communication (RS232 or RS485) between the computer and the electronic and embedded controller driving the motors. They use the same physical communication layer but different protocols. We can immediately see the object candidates, from low level to high level: Basic serial communication, robot communication protocol, rotating table communication protocol, robot itself and rotating table itself.

The basic serial communication will simply drive the computer serial port to send and receive characters. It has no knowledge of what represent the data sent or receive. It only knows how to send data and receive data.

Communication protocol object handle messages required to instruct a robot controller or a rotating table motor controller to do what it needs to. This object doesn’t know how to send characters to a serial port. It even doesn’t know it is making use of a serial port. But it knows how to format a message instructing the robot or the table to reach a given position. The communication protocol object has no idea what is the movement purpose, but it know how to request such movement.

Robot or rotating table objects are high level objects. They know what a robot or table is able to do, they know which sequences of instructions are required for everything a robot or table can do, at least from a low level point of view. The error to be avoided here is to put in that object something related to yet a higher level. That higher level is related to the coordination between the robot and the table, or related to the whole systems.

I have just scratched the surface of the OOP design for such an application. I don’t want to teach you how to build software for automated digital radiography system. I want to teach you how to write good Delphi code (Well, this applies to almost all object oriented programming language).

An object in the middle of a hierarchy receive order from the higher level and has to delegate work to the lower level objects. Those are simple method calls. For example, the robot having to move the arm to a given X, Y, Z position is space (You also need the 3 angles to fully define a position in space) will call a bunch of methods of the communication protocols, probably a message to send each parameter (coordinates and angles) to the robot controller. The communication protocol object will build messages, adding addresses, message number, checksum or CRC and similar items required to make a valid message. It will delegate the sending of the message to the lowest level object handling serial port communication. This object doesn’t know what the messages are but knows how to send it one byte or character at a time, with proper baud rate, parity, start and stop bit, and how to handle handshaking.

So far so good: we only used simple method calls up to now. We are now at the point where we need even driven programming!

The software is driving hardware. Moving a robot arm or rotating a table takes a huge time compared to the computer processing speed. What happens when the requested movement is done or when something went wrong? There is a data flow in the reverse direction! The hardware (Motor controller) sends a message thru the serial port to say – for example – the position has been reached.

The lowest level object receives bytes from the motor controller. After checking errors such as parity, it transmits those bytes to the communication protocol object which assembles complete messages. Messages are checked for validity (format, length, CRC, and so on). Complete messages are then used to notify the robot object or table object that the requested movement is done. It is likely that a single movement is part of a group of coordinated movement. The robot object knows about this coordination and collects all messages until the group is finished and only then forwards the information to the upper layer.

In an event driven software, the backward information flow is handled by events. This means that the software send a request for something and don’t care waiting until the requested operation is done. Rather, it handles notification when something happens, for example the end of requested operation or an error message. This event driven operation is frequently called asynchronous operation because requesting something is decoupled from waiting for it to be done.

Traditional programming, also known as synchronous programming or blocking programming, works by sending a request and waiting for the answer. This is simple and easy. Well easy until you have to do several things simultaneously. With synchronous programming you must then use multithreading to do several thing simultaneously. This works well but it is difficult to develop, debug and maintain. There are a lot of issues arising from thread synchronization. This is a very difficult matter.

Asynchronous programming or event driven programming solves those issues easily. There is no problem at all doing several things simultaneously since “things” are merely requests to do something. The request is almost instantaneous. There is no wait, no blocking. The program never wait that something is done. It just does the processing when it is done.

Think about Windows user interface. Your code never wait that the user clicks on a button. You just assign code to the event which is triggered when the user clicks on the button. The code is executed when the user clicks and you have nothing to do for that to happen.

This event driven behavior can be built into your own objects very easily. It fits very well along to the code you write for the user interface.

Here are the required steps:
  1. Think about which data you need to pass when the event is triggered
  2. Create a data type corresponding to the data found in step 1. Add a “Sender” argument. This data type is a pointer to a procedure of object.
  3. Create a protected member variable in your class to hold the data type from step 2. Usually this member variable name begins with “FOn”.
  4. Create a published property corresponding to the member variable.
  5. Create a protected “Trigger” virtual procedure.
  6.  Implement the trigger procedure
You want an example? The communication protocol object which receive data from the serial communication object will trigger an event when it has assembled a full message and this message in a message stating the position of the movement (Usually motor controller periodically send such message while moving). It is likely that the data is the actual position. This position is usually expressed as an integer count of a position encoder tick.

Step 1:
We need an integer.

Step 2:
type
  TPositionEvent = procedure (Sender : TObject; Position : Integer) of object;

Step3:
protected
  FOnXPosition : TPositionEvent;
  

Step 4:
  published
    property OnXPosition : TPositionEvent read  FOnXPosition
                                          write FOnXPosition;

Step 5:
protected
    procedure TriggerXPosition(Position : Integer); virtual;

Step 6:
procedure TMyObject.TriggerXPosition(Position: Integer);
begin
    if Assigned(FOnXPosition) then
        FOnXPosition(Self, Position);
end;

Usually I use a single source file for each individual object.
The complete code should looks like this:

unit RobotCommProtocol;

interface

uses
  Classes;

type
  TPositionEvent = procedure (Sender : TObject; Position : Integer) of object;

  TRobotProtocol = class(TComponent)
  protected
    FOnXPosition : TPositionEvent;
    procedure TriggerXPosition(Position : Integer); virtual;
  published
    property OnXPosition : TPositionEvent read  FOnXPosition
                                          write FOnXPosition;
  end;


implementation

procedure TRobotProtocol.TriggerXPosition(Position: Integer);
begin
    if Assigned(FOnXPosition) then
        FOnXPosition(Self, Position);
end;

end.

Carefully study how I named the various parts. Naming convention is very important to have readable and maintainable code. Keep naming same thing with same name, using prefixes or suffixes to make a distinction where required.

My event is supposed to return a position. Assuming we have several possible positions, I named everything related to the event “XPosition”. The data type is named “TPositionEvent” because the same event will apply to X, Y, Z and all others so the “X” has been dropped. It is to be used for an event so the suffix is “Event”. And it begins with letter “T” because it is a data type.

The property itself is named “OnXPosition” for obvious reasons. Think about the “OnClick” event of a TButton. This is similar.

The member variable has the same name as the property with an “F” prefix. This convention is almost always used.

The trigger procedure begins with prefix “Trigger” and become TriggerXPosition. When the object needs to trigger the OnXPosition event, it will call TriggerXPosition, passing the new position. The procedure is made virtual so that derived classes have a chance to override his behavior. For example, the derived class could enhance it be triggering another event when the value exceed some limit.

The unit containing the code has been named “RobotCommProtocol” because our object handles a communication protocol for a given robot. The object itself is named “TRobotProtocol” for obvious reasons. It derives from TComponent which makes possible to install the object as a component available in Delphi IDE component palette. There are other requirements which are out of this article scope.

There is much more to say about the topic. Please post a comment to the article to ask for the topics I should develop in the next article.



Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be
This article is available from http://francois-piette.blogspot.be

November 17, 2013

Upgrade option for any older version of Delphi

Currently Embarcadero offers an exceptional upgrade option for any older version of Delphi, C++Builder and RAD Studio. The discount is up to 45%, valid up to the end of year.

This is a real opportunity if you still use an old Delphi. You will really benefit upgrading your old Delphi 5, 7 or whatever to XE5. I know, I've done it. Almost all my application have been ported to the latest Delphi version.

There are a lot of articles about porting old Delphi code to recent Delphi (Mostly pre-Unicode Delphi to post Unicode Delphi). And if you think you can't do this port yourself, I'm sure you'll find a Delphi expert who will help you. I suggest you look at Embarcadero MVP directory.


Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be
This article is available from http://francois-piette.blogspot.be

November 10, 2013

Delphi uses Excel to create a chart in a PDF document

Microsoft Excel exposes all his features thru a COM interface which can easily be used from a Delphi application. In this article, I use that feature to create a 3D pie chart from data available within a Delphi program and produce a PDF document.

I already talked about using Microsoft Office applications in this article. I gave examples using Word. In this article I use Excel for what it does very well: take an array of data and produce a nice chart.

Excel exposes a number of objects and this makes programming it a little bit confusing at first. The three most important objects are:

  • ExcelApplication: this is the whole Excel application.
  • WorkBook: This is a spreadsheet file
  • WorkSheet: This is a page within a workbook.

There are a lot of other objects or object collections. In this article we will use “Cells” and “Charts”. They are exactly what their names imply.

Each object or collection has a lot of properties and methods. This is where it becomes quite complex. Although most names are explicit, their use isn’t. Microsoft publishes a lot of documentation (http://msdn.microsoft.com/en-us/library/office/bb726434(v=office.12).aspx). Of course none of this documentation is written using Delphi syntax. Nevertheless it is of great help even if most samples are VBA or C#.

There are a large number of Office versions. The programming interface change slightly between each version but all in all, upward compatibility is excellent. The gold rule is to always use the oldest API version suitable for what you need to do. Because of upward compatibility, your application will generally work for the version you selected and all more recent versions.

For my sample application, I used Excel 2010. Microsoft reference is here.

In Delphi, you must use the correct components. See discussion in this article. What I said then for XE4 is valid for XE5 as well as previous versions.

My demo application is simple: A single VCL form with a single button. The button’s OnClick handler connect to excel, create a workbook having a worksheet, fill cells with simple data, create a new chart with the data, export the chart as a PDF file, close the workbook and Excel.

I hardcoded the data to keep the code simple. It is quite trivial to fetch data from anywhere, including some database. How the data is fetched is not today’s article object.

There are a number of traps when writing this kind of application. Most Office API functions have a lot of arguments. Most of them can be left empty. When you specify some argument the code may triggers an access violation or an OLE error. For example, when adding a chart, on argument specifies the chart type. I’ve found that using it will trigger an OLE error. I had to left it empty and then change the property ChartType to actually change the type of chart. This is really annoying because error messages are not explicit at all! It is a try and error play. It is time consuming.

The resulting code is very short and simple indeed:

procedure TForm1.Button1Click(Sender: TObject);
var
    WBook  : ExcelWorkbook;
    WSheet : ExcelWorksheet;
    Row    : Integer;
    WChart : ExcelChart;
    LCID   : Integer;
begin
    // Get the locale identifier for the user default locale
    LCID := GetUserDefaultLCID;
    //Connect to Excel application, this will launch excel
    ExcelApplication1.Connect;
    // Make excel visible (This is not required)
    ExcelApplication1.Visible[LCID] := TRUE;
    // Create a new workbook with a new sheet
    WBook  := ExcelApplication1.Workbooks.Add(xlWBATWorksheet, LCID);
    WSheet := WBook.ActiveSheet as ExcelWorksheet;
    // Add some data to the sheet
    WSheet.Cells.Item[1, 1] := 'Item';
    WSheet.Cells.Item[1, 2] := 'Quantity';
    for Row := 0 to High(Data) do begin
        WSheet.Cells.Item[2 + Row, 1] := Data[Row].Item;
        WSheet.Cells.Item[2 + Row, 2] := Data[Row].Quantity;
    end;
    // Create a new chart
    WChart := WBook.Charts.Add(EmptyParam, EmptyParam,
                               EmptyParam, EmptyParam, LCID) as ExcelChart;
    // Set the chart type
    WChart.ChartType := xl3DPie;
    // Set the tab name
    WChart.Location(xlLocationAsNewSheet, 'MyChart');
    // Export the chart as a PDF file
    WChart.ExportAsFixedFormat(xlTypePDF, 'MyChart.pdf', xlQualityStandard,
                               TRUE, FALSE, EmptyParam, EmptyParam,
                               TRUE,         // Open after published
                               EmptyParam);
    // Close the workbook, quit excel and disconnect
    WBook.Close(FALSE, EmptyParam, EmptyParam, LCID);
    ExcelApplication1.Quit;
    ExcelApplication1.Disconnect;
end;
--
Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be
This article is available from http://francois-piette.blogspot.be