TrafficDigger's Blog

Brain controlled Traffic Simulation?

2012-01-07T22:54:00.002-05:00

Here is a brave wave detection device that can read your brain's EEG wave, at an amazing price less than $200.

http://neurosky.com/Products/MindSet.aspx
There is even a SDK for this device.
http://store.neurosky.com/collections/coming-soon

Wow- the demo shows this little gadget can even typing characters, controlled by your brain wave. Seems someday we can develop some brain-controlled traffic simulation. I can just be lying in bed with my eyes closed and by "concentrating" and my brain would directly control the simulation task. Brain-Driven Traffic Simulation.........

Nokia N8 刷机

2012-01-01T18:24:00.002-05:00

Nokia N8用了一年多了，保修期刚过。这是一部北美版的英文机 - 无法看中文网页和中文短信。因为下个月回国，今天决定把它的固件刷成全中文的。

1. 下载NaviFirm Plus
2. 再下载Nokia 刷机软件Phoenix from here
3. 用第二步下载的特殊版本的firmware 11.5降级。
4. 降级后，再用第二步下载的Phoenix 软件升级firmware 至25.07 (这是目前有中文持的固件的最高版本，用NaviFirm-Plus下载）。
5. 刷机后， Nokia Ovi Suite 会提示新版本25.08 - 注意！不要升级到25.08，因为这不是一个支持中文的版本。

成功！ :-)

Delphi exception, Fortran mystery

2011-10-20T10:56:00.003-04:00

Yesterday I had experienced a most mysterious exception which turned out to be caused by a simple mistake.

As I said, I have an enterprise SQL database server that provides geodata to ArcGIS. The geodata are generated by a sophisticated and powerful macroscopic traffic simulator we developed. The simulator, called DYNEV is super fast that for super large transportation networks, that for thousands of links and wide area simulation, it will take less than a minute for 10 hour simulation time with dynamic traffic assignment.   There are many gems about this simulator, its modeling, and the system but probably I'll talk about it later.

The DYNEV simulator is written/rewritten in tens of thousands lines of FORTRAN 2003/2008 with FORTRAN's latest nice and sexy features, while retaining the performance of original FORTRAN assembly-like power. The data exporter to various databases, however is written in Delphi (objecct pascal) in the format of a DLLs , because the Delphi's super power to interface with different, virtually near all kinds of databases.

Yesterday when we run DYNEV, we had a fatal exception happened, thrown from DYNEV program, towards the end of the simulation run when trying to export the data to SQL server.
The mysterious thing is, the exception only happens on deployment machine; if running from my development machine, everything is running beautifully.

It cost me quite a while to figure it out -
On the development machine, I have SQL Server Express 2008 R2, which is a capacity limited, local SQL server, installed for debugging and testing purpose. When developing, I use that local server as the default database connection, but the system has an ini file that specifies the real database server to connect.

The mistake I made is, the SQL server connection instance, when created and initialized, uses the default connection set up, which will try to create a connection in the first place to the local SQL server on my development machine. Something like this:

lvSQLConn := TMSSQLConnection.Create;

try
   try
      ...
      lvSQLConn.ServerAddress := (server address specified from the ini file)
   except
       ...
   end;
finally
   lvSQLConn.Free;
end;

So the default connection happens during the construction. Of course, when running on my development machine, there will be absolutely no problem, because that local SQL server will always be available to access as Named Pipe.   However, on deployment machine, that local SQL server becomes inaccessible (because network remote connection is disabled), hence an exception (something like "Server doesn't support requested protocol") is thrown right in the middle of construction, which, as you see from the code, is not caught by the try-except-finally, but caught by its host, i.e., DYNEV, which is in FORTRAN and has no idea about the type of exception.

The fix is very simple and straightforward, once the cause is pin-pointed:

Move the class construction line into the try-except-finally
Turned off the default connection (i.e., set the Active := False for the data access component during design time).

VPN connect and disconnect script

2011-10-18T21:32:00.006-04:00

Assume a business model uses two database servers, one is master, behind a corporate firewall; the other is a replication behind another firewall. At midnight, a one-way replication will be performed to sync the master database server with the replication one. Before the synchronization is performed, a VPN connection needs to be established, and the VPN connection will be released after the synchronization is done.

Autoit, http://www.autoitscript.com/site/autoit/ provides a very neat solution to automate this - that is, first create an Autoit script, then compile the script as an executable, and schedule the executable as a Windows task.

The tricky part is, in order to make this work, the scheduled task MUST be set as "Run only when user is logged on". This is because if not, Windows OS will run the program in the background and Autoit cannot grab the window's handle therefore any key strokes emulation will not work.

In summary,

Download AutoIt
Create a script like this:

Run("C:\Program Files (x86)\Cisco Systems\VPN Client\vpnclient.exe connect user wuppy pwd puppypwd ")
Sleep(5000)
ControlSend("[CLASS:WinConsoleClass", "","","y")
Sleep(1000)
RunWait("dbsync.exe")
Run("C:\Program Files (x86)\Cisco Systems\VPN Client\vpnclient.exe disconnect")

Use AutoIt compile the above script.
Set a scheduled task with Windows task scheduler, with "Run only when user is logged on" option selected.
Locked the user account for which the above task is scheduled. This is important. You must NOT log off because then the task will not be run (even you set it as "Run whether user is logged on or not"). The reason is the CISCO VPN client connection if running in the background (i.e., "Run whether user is logged on or not" is selected), it can NOT receive emulated key strokes to accept the connection, hence the scheduled task will be launched but it will be hanging there forever.

In the above script, RunWait command is used to launch the database synchronization program dbsync.exe. Then the script execution waits till dbsync.exe finishes, before disconnecting the VPN connection.

Also noted that "ControlSend" is used instead of "Send" to send the emulated "y" key stroke, as in the following script line:

ControlSend("[CLASS:WinConsoleClass", "","","y")

Without the emulated "y", the CISCO VPN client will simply hang there forever, waiting for the user's confirmation.

The difference between "ControlSend" and "Send" is Send is going to send an emulated key stroke to the active window, while ControlSend to any window specified with its class title. Send will not work with the logged user locked, since in that case there is no "active" window as seen by Autoit.

Note on OmniThreadLibrary (OTL): Communication Channel

2011-10-09T11:10:00.002-04:00

To facilitate communication between two thread task workers, OmniThreadLibrary provides a very handy mechanism, i.e., communication channel.   Two different task works, each can register the one end point of the channel, and communicate through that channel like this:

Taken from demo 8 RegisterComm,

In the main thread, create the communication channel, the task controls, and associate the two end points of the channel with each of the task worker:

procedure TfrmTestRegisterComm.FormCreate(Sender: TObject);
begin
FCommChannel := CreateTwoWayChannel(1024);
FClient1              := CreateTask(TCommTester.Create(FCommChannel.Endpoint1))
                                  .MonitorWith(OmniTED)
      .Run;
FClient2              := CreateTask(TCommTester.Create(FCommChannel.Endpoint2))
      .MonitorWith(OmniTED)
                                  .Run;
end;

In the task worker's constructor, keep a local reference to the end point:

constructor TCommTester.Create(commEndpoint: IOmniCommunicationEndpoint);
begin
inherited Create;
ctComm := commEndpoint;
end;

In the task worker's initializer, register the end point with its task:

function TCommTester.Initialize: boolean;
begin
Task.RegisterComm(ctComm);
Result := true;
end;

From the above, you can see, the steps are:

Create the two way channel using "CreateTwoWayChannel", the number 1024 indicates the length of the message queue;
Associate the two end points of the channel with two task workers;
Inside the task worker's Initialize( ), register the communication end point with the worker's IOmniTask.

After all these are done, and wen a task worker wants to talk to the other task worker, simply do something like below, then the other task worker's MSG_FORWARDING message handler will be invoked. Isn't that nice?

Therefore, if a task worker wants to talk to its task control, it can keep using Task.Comm.Send, or if it wants to talk to another task worker, use
ctComm.Send(MSG_FORWARDING, msg.MsgData);

In case you want a task worker to send messages to ITSELF, you can register the same communication channel "twice" with the same task worker like the following:

function TCommTester.Initialize: boolean;
begin
Task.RegisterComm(ctComm);
Task.RegisterComm(ctComm.OtherEndpoint);
Result := true;
end;

Cool!

Using Delphi DLL with C++

2011-10-05T18:29:00.003-04:00

Delphi has very good support for developing database applications. Recently I am working on an database exporter of a scientific application developed in C++/Qt. Because of Delphi's superb capability and easiness handling database, I decide to use Delphi and DevArt's DAC components. To inter-op with the C++ host, the database part is developed as a DLL.

The tricky part is, the DLLs generated by Delphi won't have import lib files, which are required by the C++ host. To overcome that, you have a few options:

Generate proper import lib by defining your own def file, see this post
Convert Delphi DLL to static lib using this tool
Or dynamically loading the DLL at run time.

To dynamically load the DLL at run time, you have to do the following tedious steps (excerpted from MSDN):

Define the function prototypes.

typedef short (CALLBACK* FindArtistType)(LPCTSTR);

Load the DLL using the LoadLibrary function.
Keep the handle to the DLL instance.
dllHandle = LoadLibrary("art.dll");
Get a pointer to each function using the GetProcAddress function.

Cast function pointers to the types defined in Step 1.

FindArtistPtr = (FindArtistType)GetProcAddress(dllHandle,
         "FindArtist");

Verify function pointers.

if (runTimeLinkSuccess = (NULL != FindArtistPtr))

Use functions.
short retVal = FindArtistPtr(myArtist);
Unload the DLL.
freeResult = FreeLibrary(dllHandle);

Qt provides a very nice class, QLibrary to streamline the above procedures, and the nice thing is, the instance is ref counted so if a DLL won't be unloaded unless all QLibrary instances go out of scope.
The following is the sample code that loads the dll developed in Delphi. Isn't that nice?

Blog Statistics

2011-09-11T13:39:00.004-04:00

I am really perplexed by the statistics of this blog. It indicates that the post "Interfacing AIMSUN API with Delphi" has the highest number of views.

I am wondering how many Aimsun users are using Delphi instead of C++ for developing externa apps?

VISSIM copy protection and others

2011-09-11T11:12:00.008-04:00

It is very interesting that this blog has some traffic directed by key words such as "vissim hack", "vissim patch" etc.

Of course, this is not a hacker's blog, while, I believe ever since VISSIM employs WiBu CodeMeter as its DRM, it is IMPOSSIBLE, or near IMPOSSIBLE to hack VISSIM or what so ever.

Honestly, I don't see the point of hacking VISSIM since it is a profession software; its end users are traffic and transportation consulting business, public sectors or academics. It is beyond my imagination that they would use a hacked VISSIM to do their projects. Granted that VISSIM licenses are not cheap but comparing to projects with hundreds of thousands of dollars budget, licensing fees are fairly trivia and in some cases are even paid off by the clients . Besides VISSIM also offers affordable academic licenses.

Oh yeah yeah yeah yeah, you are a dedicated, stubborn, enthusiastic, hard-core hacker with tons of free time and no worries for making a living, and sure you want to prove yourself - well, then see below

--------------------------------------------------------------------

WiBu each year has a hacker's contest and so far nobody is able to claim its $15000 prize yet.

Chinese Hackers Fail to Crack CodeMeter’s Strong Security

No takers for $15,000 prize; Wibu-Systems survives yet another hacker’s contest

SEATTLE, WA – (Dec 12, 2010) – Among the 138 Chinese software developers, students, professors, and other high-tech gurus who applied to crack Wibu-Systems’ CodeMeter software protection system, no one was able to collect the $15,000 prize.

Registered applicants were given a protected software application, a CodeMeter hardware CmStick, and four weeks to crack the application. Just as in previous contests, no one was able to crack the CodeMeter-protected application. The task was to create a version of the application that would run without the CmStick being present in the computer. At stake was a prize of 100,000 RMB or about $15,000.

Zou Haiyan, National Sales Director of Wibu-Systems (Shanghai), explains: “It is a big success having proven the high security level of CodeMeter in our Hacker’s Contest in China this year. Of course it is impossible to have 100% security. But with CodeMeter our customers will get a very high level of security. We constantly improve our security issue so that our customers can always rely on a strong protection solution.”

CodeMeter by Wibu-Systems AG, headquartered in Karlsruhe, Germany, is a proven and established hardware and activation-based solution protecting and licensing software used by software vendors worldwide. Continual improvements in encryption/decryption technology provides CodeMeter with strong mechanisms against cracking, for example record/playback attacks. Inside the CodeMeter smart card chip different security algorithms are used: 128-Bit AES, 224 Bit elliptical curve, and 1024 Bit RSA. Software vendors can decrypt only the part of the software that is running so that never the complete software is available unprotected on the computer. In addition, all communications between the CodeMeter stick and the software application are encrypted to prevent so-called “man in the middle” attacks.

Build log4cxx with Visual Studio 2010

2011-09-10T22:22:00.005-04:00

Steps for building log4cxx with VS2005 and VS2008 can be found from here
http://www.dreamcubes.com/b2/software-development/28/log4cxx-for-win32-with-vs2005/

And to build log4cxx with Visual Studio 2010, some steps are advised here
http://permalink.gmane.org/gmane.comp.apache.logging.log4cxx.user/2842

These steps are simply copied below in italics fonts for convenience:

1. Move all those LOG4CXX_LIST_DEF macros to before the class definition. in telnetappender.h you will also need to move the typedef that precedes this macro.

2. If the compiler complains about KeySet not being member of LoggingEvent, just remove the scope (since we moved the type to outside the class in the previous step, these types no longer are inside the class) ex: change: LoggingEvent::KeySet set; to: KeySet set;

3. If the compiler complains about insert_iterator not being in the namespace std, add #include to the include section of the source file

4. For some reason the VS2010 converter 'forgot' to reference the lib files from apr,apr-util and xml, so i had to add them manually.

I followed these steps and have the following observations:

Step 4 didn't happen to me.
Step 1 is a little complex depending on what compiler you use. If you use Visual Studio 2010's default VC++ compiler 10, then you'll get error C2252 : an explicit instantiation of a template can only occur at namespace scope; however, if you use Intel C++ compiler 2011 SP1, you'll just get a warning. After moving the LOG4CXX_LIST_DEF macros to before the class definition, Intel C++ compiler 2011 SP1 will drop the warning but pop out a new warning about explicit instantiation more than once. In other words, with Intel Compiler, Step 1 is optional (as I've tested, you can just simply ignore these cute warnings).
Step 2 is necessary for both VC++ and Intel C++ compilers
Step 3 is necessary for both VC++ and Intel C++ compilers

Interestingly, log4cxx built with Intel C++ compiler with some Intel specific optimization switches on shows some what performance improvement by 5% ~10% (this is a rough estimate based on the before-and-after comparison of some program I wrote, you mileage may vary depending on the context, use case and what particular switches).

Also Intel C++ tends to generate larger binary files.

After the play with myself on this, I organized the log4cxx Visual Studio 2010 solution files (with additional settings for Intel compilers). It can be downloaded from here , so you don't have to go through the troubles yourself.

If you download it and find it helpful, please let me know and say "Thank you!" :-)

Note on OmniThreadLibrary (OTL): ChainTo/MsgWait

2011-08-19T13:07:00.003-04:00

ChainTo is a very handy decorator of IOmniTaskControl interface - the use case is

[we have a list of tasks, and we want to assign these tasks to a list working threads, one task a thread, and have these threads executing one after another, sequentially]

For example (Sample 16),
procedure TfrmTestChainTo.btnStartTasksClick(Sender: TObject);
var
task1: IOmniTaskControl;
task2: IOmniTaskControl;
task3: IOmniTaskControl;
taskA: IOmniTaskControl;
begin

task3 := CreateTask(BgTask, '3').MonitorWith(OmniTED);
task2 := CreateTask(BgTask, '2').MonitorWith(OmniTED).ChainTo(task3);
task1 := CreateTask(BgTask, '1').MonitorWith(OmniTED).ChainTo(task2);

task1.Run;

taskA := CreateTask(BgTask, 'A').MonitorWith(OmniTED).ChainTo(
CreateTask(BgTask, 'B').MonitorWith(OmniTED).ChainTo(
CreateTask(BgTask, 'C').MonitorWith(OmniTED)));

taskA.Run;
end;

When task1.Run called, task1 runs first, after it finishes, task2 starts then task 3.

MsgWait

MsgWait is needed if the worker thread calls something that involves windows message loop. For example, a timer .

FHelloWorld := CreateTask(THelloWorld.Create(), 'Hello, World!')
.MonitorWith(oeMonitor)
.SetTimer(500, MSG_INTERNAL_TIMER)
//.MsgWait
.Run;

MsgWait is not needed for SetTimer, but needed for the following:

function THelloWorld.Initialize: boolean;
begin
FTimer := TDSiTimer.Create(true, 1000, DoTimer);
Result := true;
end;

The following is exercpted from The Delphi Geek Blog:

[http://www.thedelphigeek.com/2008/09/processing-windows-messages-in.html

MsgWait

In case you want to learn more about OTL internals, read ahead …

Let’s take a short look at the .MsgWait implementation. The function itself just sets two internal fields and returns the object itself so that we can chain another method to it.

function TOmniTaskControl.MsgWait(wakeMask: DWORD): IOmniTaskControl;

begin

 Options := Options + [tcoMessageWait];

 otcExecutor.WakeMask := wakeMask;

 Result := Self;

end; { TOmniTaskControl.MsgWait }

The hard work is done in TOmniTaskExecutor.Asy_DispatchMessages. If the tcoMessageWait option is set, the MsgWaitForMultipleObjectsEx will also wait for Windows messages (in addition to everything else it does) because it will receive non-null waitWakeMask. When a message is detected, the code will call ProcessThreadMessages method which simply peeks and dispatches all Windows messages (and Delphi’s internal message dispatch mechanism takes care of all the rest).

if tcoMessageWait in Options then

 waitWakeMask := WakeMask

else

 waitWakeMask := 0;

//...

awaited := MsgWaitForMultipleObjectsEx(numWaitHandles, waitHandles,

 cardinal(timeout_ms), waitWakeMask, flags);

//...

else if awaited = (WAIT_OBJECT_0 + numWaitHandles) then //message

 ProcessThreadMessages



procedure TOmniTaskExecutor.ProcessThreadMessages;

var

 msg: TMsg;

begin

 while PeekMessage(Msg, 0, 0, 0, PM_REMOVE) and (Msg.Message <> WM_QUIT) do begin

   TranslateMessage(Msg);

   DispatchMessage(Msg);

 end;

end; { TOmniTaskControl.ProcessThreadMessages }

The Asy_DispatchMessages is probably the most complicated part of the OTL (once you understand the lock-free structures inside the OtlContainers ;) Once you understand how it works you’ll be fully prepared to write custom thread loops in highly specialized threaded code. But don’t worry, you can use OTL even if you don’t understand the magic hidden inside.

Note: The above is from the Delphi Geek Blog by Primoz Gabrijelcic, the Delphi Guru and developer of the great OmniThreadLibrary.

]

Note on OmniThreadLibrary (OTL): Task Group

2011-08-18T14:27:00.003-04:00

As we have noted in a previous post that a task group can be used to "group" control a group of thread tasks.

The interface is declared as follows:

IOmniTaskGroup = interface ['{B36C08B4-0F71-422C-8613-63C4D04676B7}']
function GetTasks: IOmniTaskControlList;
function Add(const taskControl: IOmniTaskControl): IOmniTaskGroup;
function GetEnumerator: IOmniTaskControlListEnumerator;
function RegisterAllCommWith(const task: IOmniTask): IOmniTaskGroup;
function Remove(const taskControl: IOmniTaskControl): IOmniTaskGroup;
function RunAll: IOmniTaskGroup;
procedure SendToAll(const msg: TOmniMessage);
function TerminateAll(maxWait_ms: cardinal = INFINITE): boolean;
function UnregisterAllCommFrom(const task: IOmniTask): IOmniTaskGroup;
function WaitForAll(maxWait_ms: cardinal = INFINITE): boolean;
property Tasks: IOmniTaskControlList read GetTasks;
end; { IOmniTaskGroup }

The most interesting interesting interface functions are listed below -

function RunAll: IOmniTaskGroup;
procedure SendToAll(const msg: TOmniMessage);
function TerminateAll(maxWait_ms: cardinal = INFINITE): boolean;

For example (sample 15)

procedure TfrmTestTaskGroup.btnStartTasksClick(Sender: TObject);
var
i: integer;
begin
FTaskGroup := CreateTaskGroup;
for i := 1 to 10 do
CreateTask(TMyWorker.Create()).MonitorWith(OmniTED).Join(FTaskGroup);
Log('Starting all tasks');
FTaskGroup.RunAll;
end;

and

procedure TfrmTestTaskGroup.btnStopTasksClick(Sender: TObject);
begin
if assigned(FTaskGroup) then begin
FTaskGroup.TerminateAll;
FTaskGroup := nil;
Log('All stopped');
end
else
Log('Nothing to stop');
end;

Note on OmniThreadLibrary (OTL): TerminateWhen

2011-08-18T13:26:00.005-04:00

Once a worker thread has been launched, there are a few ways to terminate it:

IOmniTaskControl.Terminate

This will set the termination signal, and if it is parameter-less, then the worker will wait indefinitely till the thread finishes normally. If a wait-out time is specified, the work will wait until the wait-out time, then kill the thread forcefully, if it has not finished.

If a thread is killed forcefully, there might be resource leak (Cleanup and destructor may not get invoked).

IOmniTaskControl.TerminateWhen

Use an event, or an IOmniCancellationToken, for example (sample 14),

procedure TfrmTestTerminateWhen.btnStartTasksClick(Sender: TObject);
var
i: integer;
begin
if assigned(FCounter) and (FCounter.Value > 0) then
btnStopTasksClick(Sender);
FCounter := CreateCounter(10);
FTerminate := CreateOmniCancellationToken;
for i := 1 to FCounter.Value do begin
Log(Format('Task started: %d',
[CreateTask(TMyWorker.Create()).TerminateWhen(FTerminate).WithCounter(FCounter).
MonitorWith(OmniTED).Run.UniqueID]));
end;
end

Then, somewhere in the code, call

FTerminate.Signal

This is equivalent to calling IOmniTaskControl.Terminate, i.e., it will set the termination signal and wait indefinitely till the thread finishes. When the termination signal is set, IOmniTask.Terminated will be True

If the thread is currently inside an intense looping, we can use the following methods to jump out of the looping immediately

[method1]
while True do
begin
...
if WaitForSingleObject(task.TerminateEvent, 0) = WAIT_OBJECT_0 then
Break;
...
end

or
[method2]
while True do
begin
...
if task.Terminated then
Break;
...
end

or

[method3]
while True do
begin
...
if task.CancellationToken.IsSignalled then
Break;
...
end

To use the last one, we need to associated a cancellation token as follows:

CreateTask(TMyWorker.Create())
.TerminateWhen(FTerminate)
.CancelWith(FTerminate)
.WithCounter(FCounter)
.MonitorWith(OmniTED)
.Run

Note on OmniThreadLibrary (OTL): Exception

2011-08-16T12:54:00.011-04:00

When an exception happens from a working thread, OTL allows to pass the exception information to the calling thread (i.e., the one where the relevant IOmniTaskControl is created). (See OTL sample 13)

The information is saved in IOmniTaskControl.FatalException, it is a normal Exception sub class, therefore if it is not nil, then from its class name and message, one can get the exception information. For example,

if Assigned(aTaskControl.FatalException) then
lvExInfo := aTaskControl.FatalException.ClassName + ': ' + aTaskControl.FatalException.Message;

In sample 13, the decorator Enforced(True) is used, i.e.,

CreateTask(TExceptionTest.Create(Sender = btnInitException)).Enforced(True)

According to the author -

"...In short – OTL always tries to execute your task. If you call taskControl.Terminate before the task has even started, OTL will set the termination signal and start executing task. This is not a good idea if the task was waiting in the thread pool queue and threadPool.CancelAll or threadPool.Terminate was executed. To bypass this auto-execute behaviour, you can call .Enforced(false)."

Given the above statement, Enforced(True) is not required, since by default it is already set to true. Therefore, sample 13 runs the same if we remove Enforced(True).

Some special remarks for exceptions that happen with TOmniWorker instances -

When an exception occurs, be it from inside a worker object's Initialize( ), Cleanup( ), or an event handler for an internal timer, the worker thread will be immediately terminated (by its task control); it is after the termination, the exception information is assigned to its task control's FatalException property, and available from there.

In all, I am not quite sure about the rationale of this behavior, but I guess the author's intention is to pass the exception information to the task control in the first place, though, before that, some necessary handling must have been performed (at the user's responsibility) in the worker thread where the exception is raised before passing that information. Otherwise, I would expect high possibility of resource leak since after the exception the work thread will be terminated unconditionally and depending on where the exception happens, Cleanup( ), and/or the worker object's destructor may not even get invoked.

Note on OmniThreadLibrary (OTL): Lock

2011-08-16T12:33:00.008-04:00

These series posts on OmniThreadLibrary (OTL) only serve as my personal mnemonics notes on the usage of OTL. OTL's internal workings are quite sophisticated, as documented in several scattering posts by the original author http://otl.17slon.com/tutorials.htm

For thread synchronization, OTL implements Spin Lock, and Ticket Spin Lock. More information on Spin Lock from here, and Ticket Spin Lock from here.

Under OTL framework, a lock can be associated with a IOmniTaskControl by

function WithLock(const lock: TSynchroObject; autoDestroyLock: boolean = true): IOmniTaskControl; overload;

function WithLock(const lock: IOmniCriticalSection): IOmniTaskControl; overload;

For example (excerpted from OTL sample 11)

procedure TfrmTestLock.btnTestLockClick(Sender: TObject);
var
task : IOmniTaskControl;
iRepeat: integer;
begin
task := CreateTask(TestArray);

if sender = btnTestLock then
task.WithLock(TTicketSpinLock.Create);

task.Run;

for iRepeat := 1 to CTestRepetitions do
if not OneTest(task.Lock) then
begin
Log(Format('Test failed at repetition %d', [iRepeat]));
task.Terminate;
break; //for iRepeat
end;
task.WaitFor(INFINITE);
Log('Completed');
end;

The autoDestroyLock by default is true, meaning the reference count will be decremented when the omni task control goes out of scope.

Note on OmniThreadLibrary (OTL): Thread Pool

2011-08-15T17:13:00.010-04:00

OmniThreadLibarry (OTL) is a Delphi-based threading library developed by Delphi guru Primoz Gabrijelcic. It is a great gift that Primoz contributes to the Delphi community (free with source code http://otl.17slon.com/index.htm). The library is very handy except currently somewhat missing systematic documentation/tutorial, therefore a user would have to go through the supplied samples.

Here is a little mnemonics on OTL's thread pool interface, IOmniThreadPool, based on Primoz's sample 11.

1. To create a thread pool, use the utility function

function CreateThreadPool(const threadPoolName: string): IOmniThreadPool;

Var
FThreadPool : IOmniThreadPool;
...
FThreadPool := CreateThreadPool('SignalOptimizationThreadPool');

A meaningful name can be set for the thread pool via the argument.

2. After creating an interface, some properties need to be configured (if not, default values will be automatically used).

MaxExecuting

Maximum number of working threads that are allowed to be executing concurrently. It can be any number up to 60. If it is set to 0, NO tasks will be executed.

MaxQueued

Maximum number of queuing threads that are allowed to be waiting.

IdleWorkerThreadTimeout_sec

The "time out" for idling threads, i.e., the idling time before the thread in question is removed from the thread pool.

UniqueID

Unique ID of the thread pool. For example,

Log(Format('Thread %d destroyed in thread pool %d', [threadID, pool.UniqueID]));

WaitOnTerminate_sec

The waiting time after a termiated is true. If the thread cannot finish its task then after this time the thread will be killed forcefully.

3. To associate a thread pool with tasks, use IOmniTaskControl.schedule

CreateTask(THelloWorker.Create(Handle,delay_ms)).MonitorWith(OmniTED).Schedule(FThreadPool);
or
CreateTask(THelloWorker.Create(Handle,delay_ms)).MonitorWith(OmniTED).Schedule;

The second case, having no thread pool explicitly specified, will use a default GlobalOmniThreadPool automatically.

For unobserved tasks (i.e., the task controls are not assigned to explicit variables, nor a task monitor is associated), we can use

FObservableTask := CreateTask(THelloWorker.Create(Handle), 'observed task').OnTerminated(ReleaseObservableTask).Schedule(FThreadPool);

where, ReleaseObservableTask is defined as:

procedure TfrmTestOtlThreadPool.ReleaseObservableTask(const task: IOmniTaskControl);
begin
FObservableTask := nil;
end;

To learn more about Unobserved, see Primoz Gabrijelcic's original post:
http://www.thedelphigeek.com/2009/11/omnithreadlibrary-patterns-task.html

4. To terminate an individual task, we can either call IOmniTaskControl.Terminate, or IOmniThreadPool.Cancel(taskID). The latter will wait for a WaitOnTerminate_sec time; after that, the thread will be forcefully killed if it has not been finished normally. The former, similarly, you can either specify a (non-zero) customized wait time, leave it parameter-less (meaning wait indefinitely), or terminate immediately (by specifying 0 wait time).

5. IOmniThreadPoo.CancelAll can be used to cancel/terminate all tasks with the thread pool. For example,

GlobalOmniThreadPool.CancelAll,
or
GlobalOmniThreadPool.CancelAll(1000).

In the latter example a wait out time is specified.

6. Sometimes we wish to wait for all tasks to finish before moving on to next action, in this case, we can associate the thread pool with a task group

for example,

waitGroup := CreateTaskGroup;

for iTask := 1 to 6 do
CreateTask(THelloWorker.Create(Handle, 1000)).Unobserved.Join(waitGroup).Schedule;

while not waitGroup.WaitForAll(50) do
begin
Application.ProcessMessages;
end;

7. Events

The following events can be associated with a IOmniEventMonitor.

OnPoolThreadCreated
OnPoolThreadDestroying
OnPoolThreadKilled
OnPoolWOrkItemCompleted

For example, if we have the following worker,

THelloWorker = class(TOmniWorker)
strict private
FDelayTIme : Integer;
FFormHandle : THandle;
FTaskID : Int64;
public
constructor Create(aFormHandle:THandle;aDelay_ms:Integer = 1000);
destructor Destroy; override;
function Initialize: Boolean; override;
procedure SleepTask(var aMsg: TMessage); message MSG_SLEEP;
procedure Cleanup; override;
end;

If we start the task by
CreateTask(THelloWorker.Create(Handle,delay_ms)).MonitorWith(OmniTED).Schedule;

Then the global GlobalOmniTHreadPool is associated with the task. Through the life cycle of the THelloWorker, the sequence of events is as follows

THelloWoker.Create -> OnPoolThreadCreated -> THelloWorker.Initialize-> ... (if Terminate is called with a generous wait out time ) -> THelloWorker.Cleanup-> THelloWorker.Destroy-> OnTaskTerminated->OnPoolWorkItemCompleted-> after IdleWorkerThreadTimeout_sec, OnPoolThreadDestroying

If a terminate (or cancel) is called with less then sufficient wait out time, then the thread is killed forcefully. In this case, the flow is as follows

THelloWoker.Create -> OnPoolThreadCreated -> THelloWorker.Initialize-> ... (if Terminate is called with a stingy wait out time ) -> OnPoolWorkItemCompleted-> OnPoolThreadKilled

integer overflow trick

2011-05-27T10:48:00.003-04:00

Delphi has a compiler switch {$Q}, if turned off the compiler will ignore integer overflow. The following is a little trick to take advantage of this switch.

Assuming you have a heavy loop that you need to obsessionally check some status, say, verify the current time clock, if certain conditions meet, break out of the loop.

You have a integer variable numLoops, the following is taken from OmniThreadLibrary by Delphi Guru Primoz Gabrijelcic

repeat
{$Q-}Inc(numLoops);{$Q+}

if numLoops = 0 then
begin
time := DSiTimeGetTime64;

if time > endTime then
break; //repeat
end;

Inc(counter);

if BaseStack.Push(counter) then
Inc(numPushed)
else
begin
Inc(numSkipped);
DSiYield;
end;

until false;

By using the trick, as you see in the above, the following code is saved:

if numLoops = High(Word) then
numLoops = 0;

Hence a couple of CPU cycles saved (this code is for benching marking a stack data structure so CPU cycles counts!)

Lessons on floating point

2011-05-25T12:28:00.005-04:00

I have been working on some work implementing Dynamic Traffic Assignment (DTA) and really encountered some issues related to floating point computations - and here is a summary of the lessons or experience -

Some of the code is in FORTRAN 2003, some of the code is in C++. Both are latest Intel compilers.

1. Intel compilers tend to generate binary code with larger size (with the same optimization switch) than Microsoft C++ compiler;

2. FORTRAN 2003 is a whole new language and shouldn't be treated the same as FORTRAN77;

3. In order for DEBUG build and RELEASE build to have identical floating point outputs,

- FORTRAN compiler->Floating Point Model must be "Strict"
- Floating Point Speculation must be "Off",

otherwise the two built versions may generate slightly different floating point numbers and if accumulated iteratively may cause non-trivial difference.

4. FOTRAN->Run-time->"Check Array and String Bounds" and "Check Uninitialized Variables" are very useful flags especially the code has to deal with legacy FORTRAN 77 code.

5. FORTRAN->Floating Point->Floating Point Exception Handling, enable "Underflow gives 0.0; Abort on other IEEE exceptions", so that floating point exceptions can be thrown timely. But I don't find a comparable switch from Intel C++ compiler.

Another lesson that calls for attention is floating point overflow - of course there are a whole bunch IEEE definitions on these, but practically speaking, this type of exception is pretty tricky to catch as I found they don't get thrown immediately when it happens (with the Intel compiler), if you have a multi-threaded program it is even harder to trace when and where a floating point overflow is triggered.

In the Dynamic Traffic Assignment work, the testing network is a tremendously huge network and the path cost itself is a big number, this big number when got exponential-ed, easily blows off the upper limits of a double precision.

1. Be careful in using the exp( ) function and protection code should be in position to check floating point overflow;

2. Using number scaling or other fast numerical techniques to approximate exp, if needed.

3. Some C++ library can check overflow, including std::, boost::, leda:: but the problem is, the overflow can be detected only BEFORE the value is assigned to a non-temporary variable. So the check point has to be executed with each iteration and that really drags down the performance.

FORTRAN Interop with C/C++ (2)

2011-02-25T17:24:00.003-05:00

Assuming we have the following function type defined in C, i.e.,
typedef int (_stdcall * SetRelTurnFlowCbFunc)(DTAEnvironment *dtaEnv, TASession *taSesn, int iterID, int arrSize, EdgeDynaAttrRec **edgeDynaAttrArr);

This type is defined to facilitate call-back using function pointers. The actual implementation of this function is defined using FORTRAN as

FUNCTION SetRelTurnFlowCbFunc(dtaEnv, taSesn, iterID, arrSize, edgeDynaAttr_C_Arr) RESULT(f_result)
!DEC$ ATTRIBUTES STDCALL :: SetRelTurnFlowCbFunc

TYPE(C_PTR), VALUE, INTENT(IN) :: dtaEnv
TYPE(C_PTR), VALUE, INTENT(IN) :: taSesn
INTEGER(C_INT), VALUE, INTENT(IN) :: iterID
INTEGER(C_INT), VALUE, INTENT(IN) :: arrSize
TYPE(C_PTR), VALUE, INTENT(IN) :: edgeDynaAttr_C_Arr

....

CALL C_F_POINTER(edgeDynaAttr_C_Arr, edgeDynaAttr_F_Arr, [arrSize])

DO n = 1, arrSize
CALL C_F_POINTER(edgeDynaAttr_F_Arr(n), aEdgeDynaAttrRec)
END DO
RETURN
END FUNCTION SetRelTurnFlowCbFunc

There are several subtleties here -
1. Use compiler directive to tell the FORTRAN compiler the calling convention. You can leave this to the default setting of the IDE you use but I always do this explicitly.

2. When converting from a C pointer to Fortran pointer, using C_F_POINTER, a explicit array size must be specified, so that the block of memory can be correctly addressed and interpreted by the FORTRAN compiler.

3. For C pointer of pointer, an inner loop is needed. In the above example, aEdgeDynaArrRec is declared as a Fortran pointer to some data structure. In the inner loop, each C pointer is individually converted to the F-pointer.

FORTRAN Interop with C/C++ (1)

2011-02-23T11:31:00.003-05:00

FORTRAN.... oops am I talking about some stone-age monster? I hope not. And the fact is that I have been recently involved in a project that requires interfacing with a FORTRAN numerical engine. Hence the need to interface a FORTRAN host exe with a C++ DLL, while, to make it more complicated, the FOTRAN host exe calls back to the C++ DLL through function pointers.

That is for sure a very interesting experience, apart from that I am getting a little nostalgia being reminded of my good old college days using FOR77 as my first programming language. That is almost about 17 years ago.

wow! I think am really getting old now, and that makes me sad.

Oh boy! 17 years after I touched FORTRAN, now I have to use it extensively, again.

TeeChart 2010 Pro and DataAbstract Installation

2010-12-02T10:03:00.005-05:00

To upgrade TeeChart 2010 std to pro, particulary, remember to remove:
1. Chart.dcu, DbChart.dcu, GDIPAPI.dcu, DGDIPOBJ.dcu, TeCanvas.dcu
2. All other Tee*.dcu

For Delphi XE pro, remove DataAbstract_DBXDriver_Enterprise_D15, otherwise, going to have error like unable to found DBXSybaseASEDriver.

VISSIM Installer flushes PATH!

2010-07-31T20:41:00.002-04:00

Obviously there is some overlooking from PTV developer(s), or whoever prepared the VISSIM installer- I notice that after installing VISSIM (5.20 SP11), the contents of PATH environmental variable are completely erased and refilled with VISSIM-related only crap!

This is annoying - really annoying... @@$@%Q#%35 as a developer you would know the PATH would contain important paths and now I have to recollect and re-enter them manually!

VISSIM Patch 103: Code Sign

2010-01-27T11:19:00.002-05:00

The hacks to VISSIM COM instance model seem working - yet now I have a practical problem. It uses code injection techniques and some anti-virus software would treat the hook DLL as malware.

In order to distribute I have to sign it with some trusted certificate. Though, I can sign it using the certificate issued to my company (I have certificate issued by GlobalSign), I really feel reluctant to do so.

So if somebody is interested in this little stuff, I can give it to you for free but, you have to code sign it with your own certificate. You can obtain a code sign certificate from VeriSign or GlobalSign.

This completes the story.

VISSIM Patch 102: Modify VISSM COM Instance Model

2010-01-18T10:35:00.004-05:00

Ha! It was a crazy weekend after TRB... and I am intrigued to come out with a neater solution for the VISSIM COM instance issue.

In the previous post I proposed to patch VISSIM.exe binary code, which turned out to be a really dirty solution.

Why?

VISSIM.exe's UI uses a .NET library while it is a mixed potpourri of native code and .NET IL code. It is never a pleasure to look at such a thing of a mixed nature - sometimes I found myself a neat freak.

Besides, this kind of patch would immediately become useless if VISSIM has any new versions.

A more general, neater solution would be to use a system level API hook technique to intercept VISSIM's call to ole32.dll API CoRegisterClassObject - the theory is simple, whatever VISSIM does internally, it has to call CoRegisterClassObject ultimately.

All we need to do is to hook this API to override REGCLS flag.

API hook is a technique used by most anti-virus software, of course, it is also a common technique by malware programmers.

I am done with this CoRegisterClassObject API hook and now I am in the middle of testing.

This little hook dll will be released soon so one can use VISSIM in the following creative scenarios previously not possible:

1. Invoke multiple VISSIM instances through COM;
2. Multi-threading COM applications using VISSIM, each VISSIM instance is dispatched in its own threading context.

Stay tuned.

VISSIM Patch 101: Modify VISSM COM Instance Model

2010-01-14T09:56:00.028-05:00

VISSIM has a licensed feature "VISSIM COM" that a user can use to do automation-style programming. It provides convenient and easy interfaces to access VISSIM functionality programmingly.

However, VISSIM COM Server by design can only be invoked in "single instance" mode. That is, if you have a COM client that tries to establish a connection with VISSIM COM server, the connection will be established ONLY to the FIRST VISSIM instance that has already been created.

In other words, a new VISSIM instance is created if and only if there is currently no active VISSIM instance running. While on the other hand, if there are multiple VISSIM instances, your COM client would only connect to the first VISSIM COM instance.

I believe there must be a good reason for this kind of design. Just like what I did with the ACDSS adaptive control system currently being implemented in New York City. ACDSS system implements a Just-In-Time simulation framework to allow a so-called real-time operator-in-the-loop mode. This JIT simulation is based on a new COM interface I developed on top of Aimsun. This Aimsun COM server allows multiple instances to be started and in-sync so the operator can use Side-By-Side (a.k.a) SxS simulations to compare.

If you are interested in the new adaptive control system, you can download this paper
"Integrated Adaptive Control with Realtime Decision Support", presented at this year's TRB just finished today.

Yesterday at TRB, a Minnesota friend asked me if it is possible to create multiple VISSIM COM server connections....

En.... that is an interesting question, and the answer lies in a clear understanding of what is under the hood of Microsoft's COM technology.

Let me do a little lecture for those non-programmers or who not familiar with the inner workings of COM.

In this particular case, VISSIM is a registered out-of-process COM local server. So I'll focus the explanation on out-of-process COM only, because in-process COM has somewhat different mechanism, so in order not to confuse I'll rather leave it to some post later.

For out-of-process COM exe server, such as VISSIM, a client program (a client program can be written in any COM compliant languages such as VB, VC++, .NET blah blah blah you name it) starts a connection by calling Windows Operating System's COM API in the following sequence:

1. CoCreateInstance
Nothing need to mention about this COM API.

2. CoGetClassObject
For In-Process COM object, CoGetClassObject will get the pointer to the class factory; but for out-of-process COM exe like VISSIM, CoGetClassObject will get the VISSIM path from the ProgID in the system registry.

Note, different versions of VISSIM has different ProgID. Also remember VISSIM is registered a a LocalServer, so no DCOM crap is involved all I am taking is within the same local machine.

3. Once the system locates VISSIM.exe, CoRegisterClassObject (this is the ultimate ole32 API to be invoked, even VISSIM uses some .NET wrappers) will be invoked in the start-up code before calling the WinMain entry function of VISSIM.exe;

Here is the signature of CoRegisterClassObject(CLSID, pUnk, dwClsContext, flags, &dwRegister)

The key point is the fourth parameter "flags". It determines during runtime whether VISSIM COM exe can be started as single instance or multiple instances.

Check MSDN for more information about this COM API:
http://msdn.microsoft.com/en-us/library/ms693407%28VS.85%29.aspx

OK, so the fourth parameter is a REGCLS enumeration variable defined as:

typedef enum tagREGCLS
{
REGCLS_SINGLEUSE = 0,
REGCLS_MULTIPLEUSE = 1,
REGCLS_MULTI_SEPARATE = 2,
REGCLS_SUSPENDED = 4,
REGCLS_SURROGATE = 8
} REGCLS;

All we need to do is to pass in REGCLS_SINGLEUSE, then VISSIM COM exe can be invoked in a "multinstance" manner.

Why? Here is the description of the REGCLS flag:

REGCLS_SINGLEUSE: After an application is connected to a class object with CoGetClassObject, the class object is removed from public view so that no other applications can connect to it. This value is commonly used for single document interface (SDI) applications. Specifying this value does not affect the responsibility of the object application to call CoRevokeClassObject; it must always call CoRevokeClassObject when it is finished with an object class.
REGCLS_MULTIPLEUSE: Multiple applications can connect to the class object through calls to CoGetClassObject. If both the REGCLS_MULTIPLEUSE and CLSCTX_LOCAL_SERVER are set in a call to CoRegisterClassObject, the class object is also automatically registered as an in-process server, whether or not CLSCTX_INPROC_SERVER is explicitly set.

Here is a little twisty - essentially it says, if REGCLS_SINGLEUSE is set, then any client connection request would result in creation of a new class object - in our case, the class object is VISSIM object represente by IVissim interface. Creating a new class object means starting a new VISSIM.exe process, because VISSIM.exe is the host of the class object. Period.

4. A little interesting additional information on what happens after CoRegisterClassObject - Internally, a I/O completion port (a.k.a IOCP) is created. You know, I/O completion port is a low level IPC mechanism. By low level, I mean, File handle, Winsock2, name pipes etc all these higher level IPC mechanism would be built on top of I/O Completion port. So IOCP is pretty ugly if you ever bother to delve into that level to satisfy your peeking eyes and curiosity.

By the way, IPC = Inter Process Communication

The created I/O completion port receives I/O completion packets then dispatches the package via Windows messaging then to the WndProc registered by ole32.dll. That WndProc then unarshall the out-of-process COM method invocation and restores the original function's stack frame, hence finishes the entire invocation COM function.

This creation of stack frame on the fly is similar to how web services are implemented. All are based on stack frames created during run time. Oh ... wait... what... you don't know what is stack frame? Stack frame is just a memory footprint to prepare a function invocation, say, you need to allocate sufficient memories from the stack, then save all the current states registers and push the parameters. Of course, in COM, the convention is always stdcall.

Depending on the threading model the subject COM objects implements (single apartment, multiple apartment or whatever), there are some nuisances but I am not going to touch it here. It is pretty complex to explain.

OK. Enough background information about the inner workings of VISSIM COM. I hope I made myself understood.

The next critical question is how to utilize this information to change VISSIM COM Server' default behavior?

The problem is, CoRegisterClassObject is called inside VISSIM.exe. Without VISSIM source code, how can we specify the REGCLS flag?

That is not a problem. As long as we know the track point is the API function CoRegisterClassObject, things are just getting straightforward.

On quick and dirty solution is to use some little reverse engineering techniques to patch VISSIM's binary code. Of course, the purpose for doing that, must be legitimately justified. In the case of my Minnesota friend, he needs this to facilitate the interoperability and integration of his system with VISSIM, the nature of which is academic research by an educational institution. Therefore according to DMCA Digital Millennium Copyright Act signed into law by President Bill Clinton in 1998, that is a legitimate cause of action generally. But don't believe what I said here coz I am not a lawyer!

Bottom line, remember, if you do so, chances are it might be interpreted as breaking license agreement hence losing technical support for real projects (Seriously, I don't really believe that is going to happen but be warned!)

You might ask, why not request a new feature from PTV? Of course one can do that and actually I would encourage anybody to do that coz that is the right channel - but sometimes I found my PTV friends are pretty busy with their own priority list so I'd prefer to do a little DIY and have some fun myself, temporarily.

Oh Geez! I am writing this on the train back to New York. With so many brain numbing sessions it is moving shakingly forward, really making me feel dizzy as if my eyeballs are falling off my face so I'll have to leave the next post tonight or tomorrow.

Stay tuned. Audience (I really hope I had some readers, but somehow I guess I myself is the only reader of my blog.... so most of the time I am just being a stupid lonely programmer mumbling to himself....whatever.... it is just something to kill time on an empty dirty train)....

Disable 8.3 Filename

2010-01-02T10:43:00.000-05:00

What is 8.3 file name? See wikipedia at
http://en.wikipedia.org/wiki/8.3_filename

How to disable it?

Windows Server 2003 and Windows XP
To disable the 8.3 name creation on all NTFS partitions, type fsutil.exe behavior set disable8dot3 1 at a command prompt, and then press ENTER.

Windows 2000 and Windows NT

To disable the 8.3 name creation on all NTFS partitions, use the following steps: Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 (http://support.microsoft.com/kb/322756/ ) How to back up and restore the registry in Windows

1. Start Regedt32.exe and locate the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
2. Select the NtfsDisable8dot3NameCreation entry.

NOTE: By default, the value for this entry is set to 0.
3. On the Edit menu, click DWORD. Type a value of 1 in the Data field.
4. Click OK and then quit Regedt32.
5. Quit Windows NT and turn off your computer.
6. Restart your computer and Windows NT.

NOTE: The change to the NtfsDisable8dot3NameCreation registry entry affects only files, folders, and profiles that are created after the change. Files that already exist are not affected.

bpl, bpi, dcp

2009-12-28T12:50:00.000-05:00

For both Delphi and C++ Builder
- bpl = dll

For Delphi
-dcp = lib header (for linking bpl) + static lib

For C++ builder
- bpi = lib header (for linking bpl)

When making a package,

Delphi generates bpl + dcp
C++ Builder generates bpl + bpi + lib (static lib)

DLL conflicting with Legion

2009-11-29T14:33:00.009-05:00

I used some "tricks" to exploit the virtual tables of Aimsun C++ objects but with the latest Aimsun 6.1 (which by default would install a Legion for Aimsun module), a previously working code segment always throws access violation exception....

It turns out that some DLL from the Legion for Aimsun module is the "bad guy".... , and the issue can be temporarily resolved by uninstalling Legion for Aimsun module.

- Following up: The cause of the crash is nailed down to the C Runtime library msvcr80.dll that Legion for Aimsun is dependent on, yet the PE loader load the msvcr80.dll of version 8.0.50727.42 which is NOT what legion expects (it expects a newer version) but I have to make it visible on system path for some reason.

- Following up again: This issue is getting more interesting because it only happens on Vista machine.

- PS, I might be wrong but Legion seems only runs on Windows platform ( didn't find that there is Linux or MacOS versions of Legion yet); this doesn't seem to be consistent with Aimsun's cross-platform implementation? (Aimsun has Max OS and Linux versions since it is developed in Qt).

VISSIM Signal Control DLL SignalGUI Flow Chart

2009-11-02T11:14:00.008-05:00

VISSIM provides quite a few interfaces for integrating external signal control logic. A user can use VAP, or, more directly, using the so-called signal control API DLL to incorporate whatever "advanced" control logic he believes.

Oh yeah, believe me, the VAP DLL actually is the same as a normal signal control external DLL, all the 3 exported functions are the same. My guess is that PTV just put an additional VAP script parser for VAP but all the underlying stuff is the same.

Also VISSIM has a RGB editor that is pretty nice but to my great surprise it was developed using .NET without any obfuscation. The Transyt 7F guys are much more wary in that they at least did a little homework obfuscating their IL code. Though obfuscation won't stop any persistent reverser with perseverance, but still it is better than nothing!

PTV guys! Honk honk! Warning! .......

I have a previous article about the inner workings of VISSIM signal control DLL. So up till now you still don't know what I am talking about, I whole-heartedly suggest you read that article first.

Togother with Signal Control DLL API, PTV also provide another GUI interface ( in a DLL) for configuring any parameters relevant to the signal control DLL. That is the so-called signalGUI dll.

The signal GUI DLL has pretty confusing control flow if you simply read its code. So I put a little more chart here to help those perplexed.

If you found it useful, for example save your life, save your marriage, or help finish your work quickly and make your boss happy, or get a raise, whatever, don't forget to drop me an email!

Aimsun COM Interface (3) - Matlab example

2009-10-31T16:27:00.004-04:00

Matlab is a very powerful computational platform that runs on multi-platforms, including Windows, Mac-OS, Linux... you name it.

On Microsoft Windows, Matlab provides pretty good support for COM automation. So today, let me illustrates a little Aimsun COM sample using Matlab.

When using Aimsun COM under Matlab environment, all the automation objects are accessed by the relevant IDispatch interface.

Here is the first line that creates a reference to the Aimsun Application object. 'angcomsvr.AimsunServer' is the progID, also in Matlab the string is single quoted, just like Pascal.

aimsunApp = actxserver('angcomsvr.AimsunServer');

After creating the reference, we need to initialize the object by:.

aimsunApp.InitAimsunApp(0);

The integer parameter 0 will be marshalled as a boolean false, indicating that Aimsun is going to be started as "non-console", i.e., GUI model.

Now that we have successfully initialized Aimsun App, we can now to whateve we like, e.g., open an network, generate new demand or run the simulation. All of these through line by line Matlab code, just like what you can usually do with Aimsun's default Python scripting.

aimsunApp.OpenNetwork('D:\Network\whatever.ang');

Or you can have a little fun by maximizing, minimizing, reposition the main Aimsun window:

aimsunApp.ShowMaximized;
aimsunApp.ShowMinimized;
aimsunApp.SetWindowPosition(0, 0, 150, 150);
aimsunApp.Run;

In order to close the Aimsun Application, use

aimsunApp.release

This will decrement the reference count and since the Aimsun COM is implemented as multi-instance COM server, decrementing the reference count by 1 will immediately drop it to 0, hence deleting the object from memory.

That is it. Next I will illustrate an Excel sample. Then I'll show more complicated simulation scenario using Pascal.

OpenMP Runtime Version Conflicts

2009-10-18T13:33:00.009-04:00

I was working on a software project that I have been working on in the past two years...

The strangest thing happened when I tried to launch the main GUI of the system - I double clicked it, it flashed in the screen then disappeared immediately.

This doesn't make sense at all - it was working just a few days ago and I didn't recall I did anything...

I launched the debugger from the IDE, but the same thing happened while none of the source code break points get hit!

My first reaction is, something must go wrong in the compiler generated start-up code....

So, Plan B.

This time I launched the OllyDBG debuggger from which I started my executable.

..... oh..... modules loaded one by one.... then...... boop!

I got this error message from a console window, and the main program just halted:

" OMP abort: Initializing lidguide40.dll, but found libiomp5md.dll already initialized.
This may cause preformance KMP_DUPLICATE_LIB _OK=TRUE to ignore this problem and force the program to continue anyway. Please note that the use of KMP_DUPLICAT_LIB_OK is Unsupported and using it may cause undefined behavioer. For more infomation,please contact Intel(R) Premier Support. "

So what is going on?

Here is an explanation from Intel website:

"Both libiomp5md.dll and libguide40.dll are Intel OpenMP Runtime library. The libiomp5md.dll is new Intel OpenMP* Compatibility library while the libguide40.dll is legacy OpenMP library. The Intel® IPP 6.x and Intel® Compiler 11.x threading libraries are switching from the legacy OpenMP* run-time library (libguide*) to the compatibility library (libiomp*).

The dynamic library, libguide40.dll, is still included in the current releases for backwards compatibility and will be removed in a future release.

The error is caused by multiple OpenMP libraries were linked in same application. For example, if a application link the libiomp5md.dll from IPP 6.x, at the same time, it also link other software, e.g third-party library, which link libguide40.dll, then the error arises because of the duplicate initialization of OpenMP Runtime library"

So it turns out that two third party libraries I was using are dependent on TWO different versions of OpenMP libraries.

In all, it wasn't a challenging problem that requires RE technique to find the cause, however it is an elusive problem as the Console warning window never show up, or it did but in a very very short time to prevent naked human eyes to notice.

Solution? For the time being I just add the environmental variable as suggested but later I am thinking of removing one of the third party libraries.

Aimsun COM Interface (2) - C# Sample

2009-07-19T12:27:00.019-04:00

As I said, I am going to post a few samples illustrating the application of the new Aimsun COM interface. And now I am showing how to use it with .NET/C#.

C# is employed for the demo, because it is the preferred .NET language. However, the procedures are pretty much the same with VB.NET, Delphi.NET, Delphi Prism, and C++/CLI.NET.

So I'll just cut long story short using C# for the illustration.

First in first, this COM server needs to be registered with Windows Operating System. This can be achieved by quite different ways. I listed some below:

Using command line parameters:

Using Win32 API CreateProcess

You can double click the link to see how to register using Win32 API. I simply copied it from another site.

Now that the Aimsun COM server is registered, the next step is to add reference to the COM server. It is pretty straightforward from Visual Studio IDE, so I'll pass it.

The following code (Click to enlarge) is the actual C# code that embodies the following actions:
1. Start Aimsun
2. Open an Aimsun network
3. Pick a replication and run - on a single step by step basis

Since this sample is only for demo purpose, I didn't put anything other than basic elements. As a matter of fact, it can be put in much more sophisticated scenarios e.g., threading, .NET remoting, or web applications using ASP.NET.

Aimsun COM Interface (1)

2009-07-19T09:57:00.028-04:00

As I have pointed out in a previous article, Aimsun is a cross-platform application that runs on Linux, MacOS and Windows. COM is a Windows platform-specific technology, so naturally Aimsun itself does not have COM/Automation support. Nor I think TSS ever wants to put that in their list of buzz words.

But sometimes, we do find COM/Automation convenient, especially when we have the need to interface Aimsun with other programming languages such as VB/VBScript, Excel, MATLAB, Delphi, FORTRAN, C#/.NET etc.

Hence the start of the development to implement a COM interface for Aimsun.

It is not a simple wrap-up of Aimsun SDK functions.

The challenge is I don't have access to Aimsun source code, neither does Aimsun's native development language, i.e., Qt/C++ seem to have a stable support for COM/Active X development, at least with ActiveQt current release.

So finally I chose to resort to Windows Operating System API and COM SDK directly. Therefore, this Aimsun COM interface is purely Window specific, and based on the following packages:

Aimsun SDK (needs special license from TSS)
Windows Operating System APIs (WIN32 API)
Windows COM SDK
A little tweak of the compiler generated C++ code and manipulation of the v-table of some C++ classes.

Currently this Aimsun COM interface has the following specifics:

Dual Interface supporting both IUnknown and IDispatch, with the latter allowing using popular scripting language VBScript, JavaScript, MatLab etc.
Earlier binding so strong typed languages such C++, Delphi, FORTRAN languages can interface Aimsun kernel without runtime performance loss.
Interop with .NET by COM-Interop/P-Invoke

This Aimsun-COM implements the following custom Interfaces, and I plan to add more if I have time:

IAimsunInstance
IAimsunDemand
IAimsunExperiment
IAimsunMicroSimulator
IAimsunMOE
IAimsunNetwork
IAimsunScenario
IAimsunSignalControl

Anyone with some experienced with VISSIM COM might find the above a little familiar.

That is right. The idea is the same, i.e., to hide un-necessary class information and publish everything as interfaces and properties, while allowing the user to access the functionality using his favorite language, be it VB, Delphi, Java, .NET or whatever language that is COM-compliant.

Another good manifestation of this idea is ArcGIS's ArcObjects.

But be reminded that .NET(/C#) interop with Aimsun-COM is via COM-interop/P-Invoke, the internal mechanism is a little different from native COM language such as C++ or Delphi.

After registration of new Aimsun COM server, the system Registry has the following added entry (Double click the figure to see):

As you can see, this Aimsun COM server has a ProgID "angcomsvr.AimsunServer", and a GUID is {04B0631C-0A6A-49E6-AB17-845A31F79B74}. The default interface is IAimsunInstance.

Keep this information in mind, because we are going to use it later with various examples.

In the following days I will demo this Aimsun COM interface with Delphi, C# and MATLAB samples. And if I have time, I am also going to show the interaction of Aimsun COM with Internet Explorer, and demonstrate some interesting (toy) web applications with it.

VISSIM Signal Control API: Let's Talk about it

2009-07-18T12:56:00.005-04:00

- This is a re-posted article from my previous one published at Blogline, with minor modifications.

Besides VAP, PTV provides Signal Control APIs for customized external signal control. To be sure, the Signal Control APIs provided by PTV include several header files and C++ source files. These files contains a plethora of information for any interested user to get some hint about the coding style and design patterns implemented in VISSIM simulator.

After studying the source codes for a few days, I am having great fun to read through the APIs and really get a grasp of the intended ideas of PTV developer(s). And, I am willing to share some hints here.

1. THE ORGNIZATION OF THE SIGNAL CONTROL API

The Signal Control APIs includes the following C++ source file

a. lsa_fehl.h/lsa_fehl.cpp: Utility functions showing various error messages are defined in these files; The messages information can be found from the string resource lsa_ra_e.rc
b. lsa_puff.h/lsa_puff.cp : Core data structures, such as T_DET, T_LSA_PUFFERare defined here.
c. lsa_rahm.h/lsa_rahm.cpp: These two files define some wrapper functions for sc_dll_functions. Mostly in German.
d. lsapuffr.h : Defines various Macros to facilitate the interfacing of VISSIM and the external controller. e. sc_dll_functions.h/sc_dll_functions.cpp : Various functions are defined here to provide read-write access to detector, signal controller, routing information.
f. sc_dll_main.h/sc_dll_main.cpp : the user mostly modifies the call back functions defined here to implement his control logic. This might involve calling the access functions defined in sc_dll_functions.h g. SC_Interface.h/SC_Interface.cpp : the interface functions that connect VISSIM and the controller are defined here.

2. MAJOR DATA STRUCTURES

The most important data structure is the sturct T_LSA_Puffer. This encapsulate Signal Controller Number, and pointers to detectors list. This struct is the building block of a LINKED LIST that contains a list of T_LSA_Puffer.

3. CONTROL FLOW

At the start of simulation, a LINKED LIST of T_LSA_Puffer will be constructed. The element of the linked list is individual T_LSA_Puffer, which corresponds to each individual signal controller, get initialized.

During the simulation, at each controller updating step, through ControllerSetValue function, VISSIM transfers a LARGE NUMBER of data, identified by the MACROS, to the signal control dll. These data is stored two places, one place the those global variables defined in lsa_puff.cpp, the other place is the LINKED LIST of T_LSA_Puffer.

It is important to NOTE, all the utility functions, and the functions defined in sc_dll_functions.h, are actually WRAPPER functions simply to facilitate the read-write of the LINKED LIST! THIS IS THE BEAUTY of the entire API package. I hope somebody can appreciate this like me.

Following this, VISSIM sequentially executes a few more call-back commands to get back the signal timing, specifically, with ControllerGetValue.

4. BIG PICTURE

The kernel of the Signal Control APIs is actually only two files , i.e., SC_interface.cpp/SC_interface.h. All the other files are wrappers that aim to organize and facilitate the signal timing calculation, read and write of various global variables.

From these Signal Control APIs and the wrapper files, we can get a grasp of the coding style of VISSIM in general and also some hints of the underlying design patters.

Overall, from the External Driver Module APIs as well as from this Signal Control APIs, it can be seen that, the philosophy of VISSM APIs is to use call back functions to expose data while the user have NO access to the internal of the program. This means, VISSIM doesn't EXPORT any functions that the user can call. All the user can to is to process the data and send back the data to VISSIM via callback again. This allows flexibility on the user side, as he can freely develop wrapper class on top of the data. But the entire process is totally transparent to VISSIM.

Interfacing Aimsun API with Delphi

2009-07-18T12:43:00.008-04:00

Note this is a re-post of my previous article published in Blogline, with minor modifications.

AIMSUN provides several interfacing mechnisms for advanced applications. Among these, AIMSUN APIs, formerly known as AIMSUN GETRAM Extensions, are widely used.

In the AIMSUN APIs manual, these APIs functions are given in terms of C/C++, while illustrative examples are also in C/C++.

I would like to mention that it is also possible to interfacing AIMSUN using Delphi (Object Pascal).

And, most importantly, by using Delphi, you can not only happily enjoy the syntactically beautiful Pascal language, but also the powerful VCL framework, and the vast selections of Delphi components from Delphi community.
For example, the great Indy component for Socket programming.

The following shows a simple AIMSUN API example written in Delphi ---- this sample can be downloaded from my web: http://www.wupingxin.net/Downloads/AIMSUN_API_DELPHI.zip , from which you can load the Delphi-generated Dll into your AIMSUN application.

The key points are to be keep in mind:

1. Use CDECL calling convention

2. Referencing a2kernelxxx.dll, where "xxx" is the version number.

3. Unlike C++, Delphi doesn't not do name mangling

library AAPI_DE;
uses
AKIProxie in 'AKIProxie.pas';
{$R *.res}
function AAPILoad:Integer; cdecl;
begin
AKIPrintString('LOAD AIMSUN EXTENSTION DLL FROM DELPHI!');
Result := 0;
end;

function AAPIInit:Integer; cdecl;
begin
AKIPrintString('\tInit');
Result := 0;
end;

function AAPIManage(time:Single; timeSta:Single; timTrans:Single; acicle:Single):Integer;cdecl;
begin
AKIPrintString('\tManage');
Result := 0;
end;

function AAPIPostManage(time:Single; timeSta:Single; timTrans:Single; acicle:Single) :Integer;cdecl;
begin
AKIPrintString('\tPostManage');
Result := 0;
end;

function AAPIFinish:Integer;cdecl;
begin
AKIPrintString('\tFinish');
Result := 0;
end;

function AAPIUnLoad:Integer;cdecl;
begin
AKIPrintString('\tUnload');
Result := 0;
end;

exports
AAPILoad,
AAPIInit,
AAPIManage,
AAPIPostManage,
AAPIFinish,
AAPIUnLoad;

begin
end.

// Pascal unit of APKProxie.h in C/C++
unit AKIProxie;
interface
function AKIPrintString(Str:PChar):Integer;cdecl;

implementation
const
AIMSUN_KERNEL_DLL = 'C:\Program Files\TSS-Transport Simulation Systems\AIMSUN NG v7.0\a2kernel7.dll';
function AKIPrintString(Str:PChar):Integer;cdecl; external AIMSUN_KERNEL_DLL;
end.

Aimsun scripting: A different perspective

2009-07-18T11:58:00.004-04:00

Aimsun is an Object-Oriented Framework and Unified Platform for Traffic Modeling. There are three fundamental key words here, Number 1 - Object-Oriented, Number 2 Framework, and Number 3 Platform

I don't know if people really understand the meaning and significance of these three words, because sometimes I really feel saddened, or, humored that people tend to identify Aimsun with other ordinary traffic simulators. Add to it that more often than not, I am scared to discover so many publications from TRB with self-claimed success in establishing whatsoever "framework".

For sure these days "Platform", "Framework" such words are squandered and are more and more becoming hype terms.

But believe me, they do bear a true meaning with Aimsun.

It is not the intention of this post to elaborate on these which are going to be the various subjects of my future narrative.

Today I would like to re-post an article of mine some time ago. It is about a different, maybe innovative perspective of using Aimsun scripting.

Aimsun has provide a plethora of programming interfaces for its users: Aimsun Simulator API, Aimsun SDK, Aimsun scripting, MicroSDK, Logistics SDK, you name it. These SDKs, together with Aimsun software and internal modeling, constitute a very well designed and architected framework.

All these APIs, albeit having different focuses, have ONE in common, i.e., they MUST be invoked within Aimsun ( i.e., Aimsun SDK and Scripting), or called-back during run time (i.e., simulator API and MicroSDK). This means Aimsun cannot be used as a COM sever, and may not be invoked by external applications while accessing its kernel functionality.

Well, it is only partially true. First, Aimsun is cross-platform, it runs on Linux, MacOS, and Windows. COM is a very much Windows specific technology, so it doesn't seem to make sense to dedicate Aimsun to a platform specific feature.

Second some the COM-like functionality can be achieved by using ANGApp class fully accessible by Python scripting. As a matter of fact, TSS makes Scripting a subset of Aimsun SDK, in terms of available functionalities.

Here, in this post, I would like to point out the feasibility of using Python C API to access the Aimsun SDK functionalties through embedded scripting. Interested readers can immediately get the hints and develop creative solutions based on this idea.

The following shows aa Python wrapper class that encapsulate some COM-like interfaces:

from PyANGBasic import *
from PyANGApp import *
from PyANGAimsun import *
from PyANGKernel import *
from PyANGConsole import *

__author__ = "Wuping Xin"
__company__ = "Traffic Digger Innovative Inc."
__copyright__ = "Copyright (c) No Rights Reserved"
__license__ = "python"
__version__ = "$1.0.0 $"
__date__ = "$Date: 2009/07/15 11:57 $"

class AimsunApp:
"Wrapper class of Aimsun ANGApp class for C++ application with Python C API"
def __init__(self, isCon = False):
self.angInstance = None
self.simulator = None
self.model = None
self.angGui = None
self.isConsole = isCon
def initInstance(self):
pass

def freeInstance(self):
pass
def openNetwork(self,filename):
pass
def runInteractive(self):
self.angInstance.run()
def __setModel(self):
self.model = self.angInstance.getModel()

def initSimulator(self,repID,simMode = 0):
.......
def runSingleStep(self):

self. angInstance.processEvents()
self.simulator.simulationStep()

Now, with this class ready, the following C++ code can call the class method from an external C++ application:

Py_Initialize();
..........
pName = PyString_FromString("aimsunpy_interface");
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);

// Build the name of a callable class
pClass = PyDict_GetItemString(pDict, "AimsunApp");

// Create an instance of the class
if (PyCallable_Check(pClass))
{
pInstance = PyObject_CallObject(pClass, NULL);
}

PyObject_CallMethod(pInstance, "initInstance", NULL);
PyObject_CallMethod(pInstance, "openNetwork","s","C:\\Test\\test.ang");
PyObject_CallMethod(pInstance, "initSimulator","i",346);
..........

Now, an interesting and expensive (cost me a few hours to find out) lesson learned is that, when trying to load a C-Extension module, i.e., *.pyd file, if the host program that embeds Python interpreter links with pythonxx_d.dll, then PyImport_Import will import the xxxx_d.pyd, even if the module's name is specified without "_d"!

A warning is in order. This approach is dependent on Aimsun's Python embedded interpreter hence runtime performance is unavoidable. But if that is not an issue, this approach would make it possible to integrate Aimsun with external applications, for example, MATLAB platform.

Inside Aimsun ANGApp.close

2009-07-17T21:58:00.011-04:00

Aimsun ANGApp is a class available to python scripting. It is a very interesting class that TSS provides to its general users, and can be very powerful if the user really appreciates how it works.

In the past a few days, I have been cleaning up my previous codes, while working on a project proposal that may require Aimsun micro + meso. And for some reason, one code snippet that has worked previously crashes on ANGApp.close ... ...

This time I am pretty hesitant to write to TSS for help - though I am sure they are going to help, as always TSS is very kind and prompt in supporting their users - sometimes a little delay if they are busy. And for these couple of days, I know they are being working their tails off on the new release of Aimsun 6.1.0 - couldn't be more busier.

So I decided not to bother them and find out myself.

All rightee ......the question is, what is really going on inside ANGAPP.close()? And why there is a crash? Let's take a partial look at the code segment:

OK, save the current state of ESI by pushing it to stack. This is a no brainer.
.text:00B620E0 push esi

Note, ECX always holds the "this" pointer for C++ class. So, after mov ESI, ECX, ESI stores the starting address of the current ANGApp instance
.text:00B620E1 mov esi, ecx

Now, since ESI+0 is the starting address of the subject ANGApp instance, ESI + 8 must be the address for some private member-variable, and by looking at the instruction at 00B620EA , we can immediately get the hint that it must be the address for a GGui* pointer . Now, what about ESI + 4? Note the fact that ANGApp inherits from QObject, thus ESI+4 actually is the address for a protected pointer of type QObjectData. What about ESI+0xC? Hey, are you as curious as I am? That is a good question! That is actually the address for a pointer of type GKModel *.

Doh~!
.text:00B620E3 mov ecx, [esi+8]

IS the GGui* NULL?
.text:00B620E6 test ecx, ecx

.text:00B620E8 jz short loc_B620FC

If the GGui* is not NULL then call GGui.getActiveModel
.text:00B620EA call ds:__imp_?getActiveModel@GGui@@QBEPAVGKModel@@XZ ;

.text:00B620F0 mov ecx, [esi+8]

Now after invoking GGui.getActiveModel, EAX holds the returned pointer for GKModel
.text:00B620F3 push eax

Then GGui.closeDocument is called with the returned GKModel pointer from above instruction
.text:00B620F4 call ds:__imp_?closeDocument@GGui@@QAE_NPAVGKModel@@@Z

Okaydokay. So far so good! Now restore ESI.
.text:00B620FA pop esi
.text:00B620FB retn
.text:00B620FC ; ---------------------------------------------------------------------------
.text:00B620FC

.text:00B620FC loc_B620FC:
[ESI + 0xC] is the address that holds the pointer of type GKModel*
.text:00B620FC mov ecx, [esi+0Ch]

Now ECX holds the pointer to GKModel
.text:00B620FF test ecx, ecx

.text:00B62101 jz short loc_B62112

Here is the tricky part. [ECX +0] is the address of the v-table of GKModel class
.text:00B62103 mov eax, [ecx]

Now EDX stores the address of the v-table of GKModel class
.text:00B62105 mov edx, [eax]

Index 1 means the first virtual function in the v-table. Note, GKModel inherits from GKObject. Therefore, the first virtual function in the v-table is the destructor ~GKObject()
.text:00B62107 push 1

Now call the destructor
.text:00B62109 call edx

Reset the GKModel* pointer to zero.
.text:00B6210B mov dword ptr [esi+0Ch], 0

This is perfect code logically showing no problem at all! And it turns out that the crash was caused by a corrupted memoy I artificially manipulated which screw up ANGApp's initernal data!

Moving from Blogline

2009-07-11T14:35:00.003-04:00

I decided to move my blog from Blogline to Blogspot ... ...

This blog will mostly document anything interesting and fun, while I am doing my work, research or daydreaming -- the subjects include but not limited to: traffic simulation modeling, programming, and reverse engineering ... ...