ChaOS Home ChaOS Source Notes ChaOS Source Index ChaOS Downloads CTPP Home
Current Diary Diary to Jun 2010 Diary to Mar 2010 Diary to Dec 2009 Diary to Apr 2009 Diary to Nov 2008
ChaOS Diary - monoblog and links to reference documents
Many golden nuggets lie herein.
31/8/2009 COBDEN MCS/CTPPNET/IP Decided to tackle an update of the Cobden MCS, which has been running the Cobden Chadwick printing machine since 1997. Tried to bring this up to date in 2003 but recompiled version crashed mysteriously so I left the old version running. The system runs on a 75MHz AMD processor, 16Mb memory and ISA architecture, so I suspected the latest versions of ChaOS use instructions which faulted the processor before the I could gain debugger control to see what was happening. Amazingly, I found the latest version of ChaOS works just fine on this old workhorse, even through CPUID instructions. But after adding the machine control modules, the mysterious crash returns - very frustrating. Eventually traced the bug to the old SVGA driver, which has now been superceded by VESA, so remove one line of code and all is well. This is exactly what ChaOS was designed for, software which can be re-used and updated even a decade after it was written.
Having re-established the ability to revise the COBDEN MCS, I can finally alter the NETUSER structure broadcasted over IPX by CTPPNET to include an IP address. Rather than rewrite all the CTPPNET code, it is a relatively simple job to embed the CTPPNET packet in a UDP datagram where a user has an IP address. On the receiver end, UDP packets incoming on port 51717 can be unwrapped and sent to the CTPPNET server code. Then, crazily, a CTPPNET broadcast sent over the internet will add a NETUSER to the remote receiver.
30/8/2009 TraceRoute/WAN IP/router ARP cache ChaOS session now running permanently on works LAN by static route to IP 82.68.176.217. Added A records to DNS server to point chaos.ctpp.co.uk and www.chaos.ctpp.co.uk to 82.68.176.217. Had a play with ICMP traceroute packet type 30 - does not seem to work, so probed Vigor router traceroute, to find only that this is done by using simple PING packets with ttl increasing from 1; routers along the way sent an error packet ICMP type 11 when a packet arrives with ttl=1 (i.e. packet cannot be forwarded). Since I am running ChaOS behind an ethernet router, the first error packet comes from the router gateway IP. The second error packet comes from the ISP internet gateway, so the router WAN IP is not visible - this is the old chestnut - How do I find the WAN IP of my router?
You can telnet your router for its WAN IP, but you need to know the correct telnet command string, so code to do this becomes very machine-specific. This is why there are so many websites out there which will report your WAN IP by returning the source IP of your HTTP session.
So I have further modified the ChaOS UDP PING (19/8/2009) to return the source IP in the return UDP->data area, which passes through routers unchanged. Now incoming packets to 82.68.176.217:51717 (UDP) with type zero (*(UL*)u->data==0) trigger a type 1 reply, which returns the source IP of the server request.
Much quicker than HTTP or telnet, gets my WAN IP in 80 milliseconds.
Playing with this, I find the Vigor router refreshes its ARP cache by sending out a broadcast to expired nodes which does not necessarily match the IP as assigned by DHCP. Modified the ARP in ChaOS to respond to any ARP broadcast with local IP.
21/8/2009 IP UDP Beginning to understand how routing works, so replaced the router at home with another Draytek Vigor 2800G. The Bind IP to MAC capability, as well as larger and more sophisticated routing tables should allow ChaOS packets to be separated from other traffic during development.
20/8/2009 IP UDP Succeeded in sending UDP packets both ways between home and on of the new fixed IP addresses at work. This was achieved at the works end (Draytek Vigor 2800G) by binding a local IP to the MAC of the ChaOS session, then setting a static route for the WAN IP into the local IP. Bind IP to MAC forces the router to assign the same IP to a given local node irrespective of the order in which systems on the LAN are booted. At home (Buffalo WBMR-G54) the ChaOS session IP is set to DMZ.
19/8/2009 IP/ICMP PING Unable to program NetGear DG384 router at work to handle the extra IP addresses at work, borrowed a Draytek Vigor 2800G. Because I have little idea of what I am doing, it took a couple of days to work out that ICMP PING packets do not pass through the Draytek router to the internet even if a local node has a Static Route set to one of the new IPs. To further confuse the issue,a PING from a Windows XP session to a Static WAN IP which is routed to a ChaOS session on the LAN does get through - presumably the router realises that the destination is within the LAN. Although I cannot find any such setting in the router setups, there must be filter to prevent PINGs travelling between LAN and internet addresses. When a PING is sent into the Draytek from the internet however, the router itself replies if the IP address is the router IP, or has been added to the WAN IP pool. So I’ve written a quick utility to send a UDP packet (source and destination ports = 51717=0xca05) out on to the internet, to see if this passes through the router tomorrow. Once the routing is established, all things become possible.
15/8/2009 IP Upgraded to 8 static IP addresses at work, in readiness for a CCTV camera server running over HTTP, so this is an opportune time to consider routing ChaOS packets over the internet to and from work. ChaOS has been able to PING across the internet for some time now, but the PING Reply packets could come from almost anywhere including the routers. Set a DMZ on the home and work routers, disabled router PING reply on both, sent a PING REQUEST to the works IP address and received a custom PING REPLY 60ms later. I know it is from the ChaOS node at work because of one particular altered digit in the reply packet. Quite exciting really, the first time ChaOS has communicated across the internet in native mode, rather than via other operating systems. Also interesting is the inbound traffic when the DMZ is on, remote IPs sending packets to try to establish TCP/UDP sessions, who knows whether these are legitimate internet traffic or viruses on the move, but of course ChaOS is an unknown species and cannot be infected by viruses designed for mainstream operating systems. Because PINGs are intercepted by many routers, they are not going to be too useful for carrying data. Most exciting would be to establish a ChaOS HTTP or FTP server, capable of handling conventional internet traffic, whilst providing an enhanced service to remote ChaOS nodes (i.e.the ChaOS source repository). Already I am having trouble keeping ChaOS versions in phase across the various development machines now running. It is logical to establish, at this early stage, mechanisms for updating source code from remote computers. The first of these is NCP, which establishes a ChaOSNET session with remote[1], compares selected source files (wildcards accepted) and prompts the user to overwrite local source files with those on the remote. It is a boon for propagating developing software across my home network for testing. But there is always scope for importing faulty source code which will break the system. So it may be time for a dedicated safe partition which holds a bootable version of ChaOS which is known to be working.
8/8/2009 ACPI Adding decoder for ACPI Methods, by creating a linked list of TermObjs at Method->child. Encountered Method object references inside Methods, some followed one or more TermArgs, which generate a parsing error; I’m guessing that these are AML function calls, as the number of TermArgs seems to match the Method template. Some are forward object references, which makes it impossible to write a one-pass decoder for ACPI which can recurse into the Methods (the number of Args for forward-referenced Methods cannot be known). So ACPI parser Pass 1 now generates the ACPI namespace, but does not recurse into Methods. Pass 2 can then create a meaningful chain of TermObjs for each Method. In practice, a simple test on the Method object can determine that Pass 2 needs to be run (ACPIOBJ->child==NULL),so if required the Method can be decoded just before execution. This saves decoding Methods which will never be used.
Method execution suite is now taking shape. By following the Method->child chain a stream of mnemonics can be displayed to understand the decode. Structure ACPICON is passed through the Method and builds the mnemonic string if ACPICON->disp is set. Next I will add ACPICON->xec, to trigger execution of the Method, and extend ACPICON to carry the execution context through to completion.
31/7/2009 D10 Laser now rebuilt and working. Not possible to use ChaOS code via serial port to control Omron C20H PLC, without writing a dual-mode serial port interrupt handler for DOS4GW (I do not want to go there!). Did have a problem with the 5V 8255 DIO relay unit, but this turned out to be nothing more than corrosion causing tracking on the VERO board used when I built this unit ten years ago. A scratch with a screwdriver to clear the corrosion and all is fine again. Holidays now for a week or two.
24/7/2009 D10 Taking a break from ChaOS for a week or two to move my laser. Will modify D10 to use Omron C20H PLC to control ancillaries, instead of the 5V solid state relays driven by 8255 on DIO card.
19/7/2009 ACPI Still working on the ACPI parser. Initial pass through DSDT DDB now creates a linked list of ACPI Objects, with the proper nesting for scope, device etc to reflect ACPI namespace. ACPI development program modified to save ACPI DDB to a file when run with no arguments; ACPI ‘.’ works on the live ACPI DDB in the current computer. ACPI file reloads a previously saved DSDT DDB, so ACPI data from different computers can be examined on the development machine. Postscript to 6/7/2009: loaded the ACPI DSDT DDB from a computer at work, to find a couple of DefStore objects in the root list. To parse these correctly, this means that the root list is a TermList rather than an ObjectList - so my note of 6/7/2009 is wrong. My reasoning so far has been drawn from the fact that all other DDBs I have examined to date have no Type1 or Type2 opcodes in the root list, so I know 6/7/2009 is in practice broadly correct. It does not make a lot of sense for ACPI to expect the OS to execute opcodes whilst loading the DDB (surely it is easier to write BIOS code to modify the ACPI data during boot-up).
On the positive side, my ACPI browser is coming on a treat. It is much easier to understand ACPI when viewing the objects in cascaded windows, with the ability to select any one to examine the contents.
6/7/2009 ACPI Returning to rework ACPI parser, to build a model of the ACPI namespace - uncovered a slight mistake in the ACPI specification. ASL (ACPI source language) grammar describes the DSDT as an ObjectList, whilst AML (ACPI Machine language - the byte stream output by the ASL compiler) says the DSDT is a TermList. It is a subtle mistake, and meaningless unless you are writing a decoder for the AML byte stream. Interpreting the DSDT as a TermList means checking for Type1 and Type2 Opcodes on the initial walk through the DSDT - but these should only appear inside an Object (a TermList is a stream of executable operations).Took me five weeks to work this out, but suddenly, although complex, ACPI now makes perfect sense.
4/7/2009 IBM eServer 445/Demo ChaOS runs fine from CD on the eServer 445 but of course until I have completed the PCI IRQ/ACPI code I can’t add drivers for the hardware, hence there is no hard disk support; it would be useful to be able to write and run test programs to investigate the hardware further, and I realise that there is already a way of doing this - using a ChaOS MULTIXEC including the OS, editor, compiler and linker, it is possible to run the whole system on a RAMdisk. This gives me the idea for an exciting ChaOS download - a CD image which self-boots, sets up a RAMdisk and extracts all the source files for the system into the development directories, demonstrating the environment used to create and modify ChaOS. Users would be able compile and run their own test programs, even recompile the whole OS. Just added a kludge to exec() to allow the recompiled version of ChaOS to be launched over the top of the boot image. Without disk support, all changes to source files are lost when the computer is switched off. For a small fee, a fully-enabled ChaOS ISOCD image (with IDE disk support) could be e-mailed anywhere in the world.
19/6/2009 ACPI Still crafting a method for extracting PCI IRQ routing from ACPI. Added UC* acpi to DEV struct to hold a pointer to the ACPI DefDevice. Next problem is to match existing ChaOS PCI bridge devices to the correct ACPI info. So far a search by name for PCI0-PCI9 or HUB0 reveals all the PCI bridges defined in ACPI. The _ADR record resolves to the PCI dev number for the bridge. _ADR may be as simple as DefDataObject <integervalue>. But on my Dell Dual Xeon the _ADR records are encoded as DefMethod, something like DefStore <value> LocalOp0 DefReturn LocalOp0 which is a pain because I have to write a robust decoder for ACPI methods to extract the address value. In the absence of a bus number info (e.g. a _BBN record) I considered that _ADR is potentially ambiguous. Bridge bus numbers can be reprogrammed, whilst PCI dev numbers cannot, so I think motherboard manufacturers would avoid wiring two bridges to the same PCI dev number, to avoid problems with PCI configuration cycles when attempting to write the contents of the bus number registers. Is this why the IBM eServer (Winnipeg bridges, all on PCI dev 0) locks up after a PCI config write to the bus number registers at PCI:0x40?
This completes the test for matching the ACPI bridges to the ChaOS DEVs. Once this matching method is encoded, there is similar amount of work to decode the PCI IRQ routing for PIC and APIC.
11/6/2009 ACPI A week of browsing the ACPI tables and a method for correctly verifying PCI IRQ routing is only just taking shape. Like any PC standard it is set out in a complex several-hundred-page PDF document, after which PC manufacturers implement the standard half-heartedly and interpret the standard in different ways. There are standard names in ACPI for a host of devices, but crucial data (such as _BBN - bridge bus number) is often missing and needs to be derived from the bridge PCI Config registers. The common thread is that this is the ONLY place in which PCI IRQ wiring is fully described, and if correctly decoded, allows complete control over IRQ routing. After carefully checking the PCI IRQs assigned by BIOS on all my development machines, the only oddball is the Cardbus slot on the Fujitsu Stylistic ST4110, which I discovered to be incorrect last year.
A Google search of PCI IRQ reports 330,000 results, PCI IRQ error reports 214,000 results, suggesting that the major operating systems struggle with PCI IRQs too!
31/5/2009 PCI IRQ routing IRQ number field at PCI:0x3c is just a placeholder for the irq which has been assigned by the BIOS during startup. I decided to rewrite the ChaOS PCI IRQ assignments in readiness for APIC interrupt handling, by using the hard-wired INTx# at PCI:0x3d. At first I couldn’t understand why the IRQs assigned by BIOS did not match the IRQs in the PCI IRQ routing registers (which were clearly working). On closer examination of the PCI local Bus specification I found that motherboard manufacturers can and do barber-pole the PCI IRQ lines for neighbouring slots, so that two identical PCI adapters using INTA# do not end up sharing the same IRQ. This is called opening a jar of worms. Further research reveals that the details of these connections are laid out in the BIOS ACPI tables, which contain a mass of device info which I have not so far tapped, yet are present on all my development machines. This ties into the MP development because the ACPI tables contain descriptors for multiple APICs. A new memory size report given by Int 0x15/0xe801 typically reserves space just below the top of physical memory for the ACPI tables.
25/5/2009 MP interrupt mapping/Dell Precision 450 Dual Xeon E7505 chipset Just discovered that IRQs, when mapped through the 8259s can interrupt ANY of the processor cores (provided their IF flags are set), which was a surprise. But it is logical, because all cores have an equal chance of winning the race to the bootstrap semaphore, and will need to be connected to the 8259s to boot the computer. I expected ChaOS to crash, watching the 4 Dell cores handling the hard disk interrupts, and I am unsure that only one core is interrupted for each hardware event in 8259 mode (clean recompile of ChaOS takes 5-6% longer, so clearly this is not efficient). Going back to APIC mode ensures only one nominated CPU core handles a particular IRQ, and best performance is achieved when that core idles in the HLT state, i.e. is not contending for other system resources whilst waiting to handle the interrupts. (PS 11.6.09 after the test running 4 cores all with interrupts enabled in 8259 mode I discovered a disk directory structure had been trashed, probably because it’s cluster number in the FAT had been set to free. This is not unsurprising as my file system code was not written with multiple processors in mind, so it is not fully re-entrant).
23/5/2009 MP multiple processors Running ChaOS .XEC test programs on the APs makes it possible to switch on their Local APICs, to catch hardware interrupts redirected by the IO APIC. Just managed to do this for the first time - BSP and APs share the same IDT (which makes the debugger work on the APs - see 17/5/2009) so hardware interrupts are inherited too. A quick check of CPUID inside the hard disk interrupt handler shows which CPU handled the interrupt. Time now to reflect for a while. Having designed ChaOS all along to be a single-task system, I never envisaged being able to do two things at the same time. But the p ossibilities of multi-threading are too exciting to be ignored, particularly since most CPUs being sold today have multiple cores. For instance a rewrite of the CO2 laser control software would allow one core to synch the laser and stepper motor pulses, which another core monitors the compressed air, extraction and cooling systems. Obviously only one CPU is mapped to a hardware IRQ at any given time (NB except in 8259 mode! see 25/5/2009), but the IO APIC could be remapped inside the interrupt handler to another CPU, to send the next interrupt to a different CPU to spread the load. Easier would be to split the hardware interrupts between the APs and leave it at that, so CPU0 can run the GUI without interruption. But to allow all CPUs access to the network and hard disks, I see no way of avoiding semaphores and spinlocks, which I have never used before. A good place to start then is probably floppy disk access, try to integrate a semaphore into the FDC DEV, and let all 4 processors fight for access, see what happens.
21/5/2009 MP multiple processors Further improved MP utility to load a standard ChaOS .XEC file, start an AP then pass command line arguments to the new thread. This requires a switch from real-mode to 16-bit protected mode, using a selector mapped to the APXECPAGE, then a jump into 32-bit protected mode useing 4Gb code selector 0x0008. A flag in the startup code indicates to MP that AP started successfully.Had a few problems setting this flag in the 16-bit protected mode section of the AP startup code - until I remembered that a code selector cannot be used to write to memory in protected mode!
15/5/2009 APIC Interrupt handling Revisiting interrupt handling via IO APIC, necessary to direct interrupts in MP environment. Had a test running last year which switched my Pentium 4 over to APIC IRQs. Processor Local APIC needs to be enabled in bit 8 of SVR, and IO APIC needs to be enabled on PCI bridge, but these bits are usually on by default. The South Bridge IO APIC usually resides at 0xfec00000, with all vectors set to 0x00000000:0x00010000 (disabled). On boot-up, IRQs are directed to the legacy 8259s, with Edge/Level control in the ELTRs at 0x4d0 and 0x4d1. Setting a vector in the redirection table, with the disable bit clear, is all that is needed to send a hardware interrupt via the IO APIC to the processor Local APIC. E.g. setting redirection vector[1] (keyboard) to 0x00000000:0x00000021 sends keyboard interrupts via the APIC bus to the processor on interrupt 0x21. The corresponding mask bit needs to be set in the 8259s, otherwise the interrupt from the keyboard is received by the processor twice (keystrokes are duplicated!). IRQ0 (timer) cannot be routed thtought the APIC (just leave it on the legacy 8259), but can be substituted by the internal APIC timer on IRQ2. Level triggered interrupts (PCI IRQs) need bit 15 to be set in the redirection vector (e.g. 0x00000000:0x0000800b), and interrupts should be directed to APs by setting the top 32 bits of the redirection vector to the Local APIC ID of the target processor (e.g. redirection vector[2] = 0x04000000:0x00000020 to send IO APIC internal timer interrupts to processor with APIC ID 4, to trigger interrupt vector 0x20).
12/5/2009 MP multiple processor support Added AP startup thunk to ChaOS, executing at address 0x18000 for APIC ids 0x01-0x1f if boot CPU shows flag 0x100 in MSR 0x1b (i.e. is identified as the BootStrap Processor, or BSP). For each AP found, a small info structure is created in the region 0x18200 and above. With a slightly modified AP thunk (AP1) it is possible to load and relocate a ChaOS .XEC program image and run it on one of the APs. Because the BSP local APIC can buffer two IPIs, no delay is necessary between the INIT IPI and the SIPI for the AP, so starting a thread takes no time. It was simple to load three copies of a program, and run each on different APs to display three counters. Obviously each AP needs a separate stack, but can share the executable image; dynamic links all work too - because all the hardware is handled by the BSP, the APs can do file i/o even though they run with interrupts disabled and have no entries in their IDTs. So ChaOS can now support multiple threads, within limits (all threads have limitations). The old RESET method of switching back to real mode is no use, as it resets all the processors and the BIOS gets confused. But executing a real-mode Int 18h or 19h on the BSP re-enters the BIOS or ChaOS bootstrap in an orderly fashion, with the APs running on regardless in protected mode! Totally wierd!
Not quite sure what to do with all this power, particularly since I designed ChaOS as a single-threaded system. Allowing multiple processors to share hardware is a challenge, but I am not really convinced that major OSes really do this - it is easier to assign processor cores to specific tasks, e.g. a network card, GUI, etc. Compilation has always been my bottleneck, and these are processor-hungry tasks could be speeded up by spreading them across the cores (e.g. load source files and pass them to separate compilers running on the APs; when a compiler completes, pass a message back to the BSP to save the object file image). This is possible because ChaOS compilation is entirely memory-based (I do not use streams for file I/O).
There seems to be no easy way of using one processor core to control another, e.g. to run an inter-processor debugger. Whilst exceptions on one processor can be forwarded to another using an IPI, this does not change the structure of the exception handlers. So single-stepping through mode switches will remain a black art. Later Intel processors have VMX, which would allow this possibility.
12/5/2009 MP Multiple processor support I have been researching multi-processor support since I bought the Dell Precision 450 in December (Dual Xeon). Because MP support involves the processor Local APICs, I have prepared a method for switching to the IO APICs to handle IRQs rather than the 8259s. However I suddenly realised that detecting and initialising multiple processors does not involve the IO APIC(s), just the Local APICs. With this in mind, it was relatively easy to write a boot thunk for the APs to see if I could wake them up on the Dell. Took a little while before I saw the magic flag set by the second processor before it crashed, but then I saw two more....(4 processors?). After another hour I realised that 'Hyper-threading' really means 'Twin-processor' and my Dual Xeon Dell contains 4 processor cores which can operate independently. Not bad for 58 pound sterling. The initialisation sequence is: 1-send global INIT (message type 5) on APIC bus to all processors (i.e write 0x000c4500 to 0xffe00300). I think this message is ignored by the BSP (boot strap-processor - well it must be because the computer does not start a reboot!). 2- send IPI (message type 6) to APIC IDs 1 to nm (i.e. write 0x000046nn to 0xfee00300, where nn is 4k memory page of boot thunk). wait 10ms after each IPI to see if magic flag is set. If so, a processor exists at that APIC ID. Simple!. Tried the same on the IBM eServer x400 and counted 8 processors. Awesome!
7/5/2009 CC2 ChaOS 'C' Compiler CC2 compiles all ChaOS OS files no problem. A pleasant surprise, as I had expected the change in language syntax to throw up some subtle bugs in existing code. The 'class' function prototypes generated by the compiler are simple copies of the member declaration with a struct* prepended to the arg list. It is a relatively simple job to generate local indirect symbols to reference all the structure data members inside the function definitions (so structure members can be referenced without the STRUCT-> notation); these symbols would be discarded after the function has passed though the compiler, otherwise the main symbol tables will become massive. But if this change in syntax was for all functions whose first argument happens to be a structure pointer, thousands of code lines would become shorter and tidier. The downside is that many such functions have arg names which are the same as structure members, often acting as member initialisers. These instances will generate errors which must be corrected by renaming the args - but a simple underscore added to the arg name fixes the error, and since I make almost no use of underscores in ChaOS symbol names, the underscore would indicate that this arg is an initialiser for the member of the same name.
6/5/2009 CC2 ChaOS 'C' Compiler Rewrite of RAM-based indexed database algorithm brings the possibilty of using these databases in ChaOS. But NJOB was written in Zortech C++, then ported to DOS4GW, and contains thousands of lines of C++. Since it probably quicker to add some C++ style syntax to the ChaOS compiler than deconstruct all that code, I decided yesterday to create a second version of the CC ChaOS compiler see if this was possible. It was easier than I had imagined (though I tried writing a ChaOS C++ compiler a few years ago and failed miserably).
C++ grew out of the concept of 'C structs with functions', so I first allow a function prototype to be defined inside a struct:
struct WOTSIT
{
UL a,b;
SL func(VD);
};
This simply generates a prototype
SL func(WOTSIT*);
After defining some structs:
WOTSIT W; WOTSIT* w=&W;
This allows the syntax
W.func(); w->func();
to invoke the 'class' function. Of course a simple 'C' invocation works too:
func(w);
Just a day later all is working fine. Revisiting this area of the compiler has resulted in improvements to the implementation of calls by function pointers too. The trickiest part was to insert a code thunk for the structure pointer in the function argument list - C calls require the args to be pushed on to the stack in reverse order. For 'class' functions with multiple arguments this means the code that references the 'class' needs to run after the code for the function arguments, with the result pushed on the stack just before the function call itself. This is the bit which defeated me a few years ago. The solution is however simple. I added two extra fields to the compiler OP structure (used when an lvalue or rvalue is created), one to hold the 'class' type, (OP->gtyp), filled in when a struct is referenced and the other to make a note of the staging buffer position in makeop(OP*) i.e. just before pcodes for the operand are generated. On reaching the 'class' function invocation, the end of the staging buffer contains the pcodes needed to address the 'class'. savestage(OP->stage) simply moves these pcodes to an extra pusharg with type=OP->gtyp. The pushargs are then passed to prototype matching to find the correct function call. If successful, the pusharg thunks are written in reverse order to the staging buffer, with the 'class' OP pushed on the the stack just before the call pcode, followed by the appropriate stack pointer adjust pcode. Simple, really!
In the absence of the above mechanism, calls by function pointer were implemented a few years ago by a workaround, simply by pushing the function pointer itself on the the stack, followed by the args, then using the x86 CALL[ESP+nn] opcode, e.g.
push pfunc
//...arg2 code thunk..., push arg2
//...arg1 code thunk..., push arg1
call [esp+8]
add esp,12
Which is a neat way of storing a function pointer for later use.
4/5/2009 NJOB Rewrite of RAM-hosted database indexing algorithm complete. Initial delta is largest power of 2 less than index length. Start pos is delta-1. If comparison <, subtract delta from pos, shift delta right by 1 and continue. If comparison >, since pos+delta may be beyond the end of the index, shift delta right while pos+delta >=top index. If delta becomes zero in this loop we have the special case, i.e record is greater than [pos], but pos+1 is beyond the index length - i.e. incoming record needs to be at the end of the index. This algorithm is in effect a hash table for a block of fixed-length records in memory - requiring the a maximum of n comparisons to to place a record in the index where the database size is up to 2^n. (As efficient as it gets). Because the tables are contiguous, the addressing is done by array arithmetic rather than using buckets containing pointers. When the database is loaded, the memory block allocated is larger than the database, providing the space for new records created during a session (the freespace). The top of the index contains the sequence of freespace locations. Deleted records are simply marked with a flag, the index above the deletion is shifted down and the deleted index rewritten to the start of freespace. This is quicker than moving the data records themselves, which would cause all index values above the deleted record to change. When writing new records, locations at the start of freespace are used first, ensuring holes in the data block are filled before increasing the length of the data. This ensures efficient REWRITEs, which usually require a DELETEKEY followed by INSERTKEY. When all freespace is used, the database is automatically saved (no bad thing when lots of records are being added) and then reloaded to create a new freespace buffer. The reload could fail if a computer has insufficient memory - (time to buy more RAM!) - but this has never happened to me. It was always the case that computer RAM capacity would grow faster than my databases.
This algorithm has been the crux of my company records for over two decades, and was originally conceived and written when I was a novice programmer - the code that resulted then was the result of a lot of trial and error. Very satisfying to distil out of it the underlying, compact algorithm (around two dozen lines of code) to take into the future.
3/5/2009 NJOB Revisiting my old RAM-hosted COBFILE indexed database code, with a view to adding indexed databases to ChaOS. Much of the code is over 15 years old, and pretty messy. There has always been a bug lurking for deleted records which disappears when the indexes are completely rebuilt (which happens each time the databases reload, at the start of the working day), so I have never bothered to tackle it. I left a note in the INSERTINDEX function in 1996 to remind me this needs fixing. The idea is to walk through the database starting at the record index which is the largest power of two, and a delta the same size which is halved after each comparison.Thus the position for the record being added is found with the minimum number of comparisons. Brief notes for the rewrite are: initial value for start = largest power of two less than i->freespace; initial value for delta is the same, except delta=i->freespace when i->freespace<2; if comparison is >, pos+=delta; if comparison is <, pos-=delta only if delta is >1 (rest of index will be shifted up by one when the new record is inserted); after comparison delta>>=1, looping as long as delta is non-zero. The above should halve the complexity of the existing code.
1/5/2009 NJOB Further to 1/4/2009, now have two business bank accounts. Modified NJOB to handle up to 9 cashbooks, simply by using the top digit of transaction number for transaction types 3 (bank receipt) and 5 (bank payment) as an account filter. Separate entries on NL170 created for the opening balance on each bank account. Cashbook (0) prints an aggregate of all bank accounts; Cashbook(1), Cashbook(2) prints Bank Account (1),(2) etc and generates separate nominal postings to bank control NL170 for each account. In this way, the starting bank balance for each cashbook can quickly be found from NL170 by testing ((transaction number/10000)==bank account number).