ChaOS Home ChaOS Source Notes ChaOS Source Index ChaOS Downloads CTPP Home
Diary to Sep 2011 Diary to Mar 2011 Diary to Dec 2010 Diary to Sep 2010 Diary to Jun 2010 Diary to Mar 2010 Diary to Dec 2009 Diary to Aug 2009 Diary to Apr 2009 Diary to Nov 2008
ChaOS Diary - monoblog and links to reference documents
Many golden nuggets lie herein. Trademarks are acknowledged as copyright of their respective owners
6/12/2011 DEC2011 ISO After a month of solid work, DEC2011 ISO is nearly ready. It features a hard disk installation routine, and HTML-based help screens, accessed by pressing the [F1] key. The principle is simple, if present, [F1] displays index.html from the current directory. Of course HTML allows any number of hyperlinks, to this can be an index into multiple help screens, as desired. The linker now accepts embedded content files, so the HTML help can be packaged in the .xec files which make up the ISO itself. ht is my new HTML browser, supporting just enough HTML tags to get this up and running. ht searches embedded content before searching the local file system for hyperlink targets - which means the help system is up and running as soon as ChaOS boots (the HTML content does not need to be extracted to disk to be seen. This is similar to the way the ChaOS debugger presents source code).
30/10/2011 RMDBG and alternate bootstrap In preparation for DEC2001 ISO download, added variation to CFS format to allow space for two bootstraps, switched in CFS bootsector on CAPS LOCK key. If primary bootstrap is broken during development, CAPS LOCK will cause system to boot from a safe backup. Should save a heap of time.
Further revised RMDBG, source-level debugging of bootstrap now working as intended. Slight problem debugging AP in real mode from INIT IPI - caused by BSP being in HLT state in getkey() - BSP is always first around the loop to grab the key, so AP hangs in the RMDBG kernel. Fixed by preventing BSP HLT when in RMDBG.
Revised boot-time keyboard state handling, now SCROLL LOCK triggers (Y/N/ESC)? prompt when attaching DRVs to PCI DEVs, to allow selective loading of PCI device drivers during startup. Adds a little more flexibility when a faulty DRV has been created.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab3: over ChaOS version 1.03.27970
29/10/2011 Hard disk FLUSH CACHE Working on new bootstrap, which involves repeated reboot shortly after recompilation of various modules. Experiencing occasional minor disk errors, fixed by recompiling the affected modules. This is only happening on my i7-920 and may be because, running at 4.4GHz, a hot reboot sometimes resets a hard disk before it has finished writing data.
Added FLUSH CACHE command to IDE and SATA drivers, called on DM_TERM or DM_SHUTDOWN, to ensure all data is written to the disks before OS restarts. Will this elusive glitch will now disappear?.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27943
25/10/2011 Stepper MOSFET Direct-Drive from parallel port Built this drive a few months ago, now that big MOSFETs with TTL-compatible gates are so cheap. The downside is that 4 lines are required for each motor rather than 2 as I am driving the windings directly, so each parallel port can handle only 3 motors instead of 6. A PCI parallel card can be had for three quid, so this is only an issue where PCI slots are limited.
Main problem with my design in that standard stripboard cannot handle the current absorbed by a medium-sized stepper motor. Any dodgy connections heat up rapidly, and tracks not reinforced with solder can be blown away. Added pull-down resistors to stop the MOSFETS from switching on when the parallel port lines are floaitng high, i.e. when the computer is switched off or booting up.
Ran 2 x MO93 Slo-Syn motors at 4000-8000 half-steps/second continuously for about 8 hours. FETs running warm, about 30 degrees C. Signal cable 5 metres, motor cables 5 and 10 metres respectively, so this idea is practical in a factory setting.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27943
20/10/2011 FTPSRV.DRV Added directory string to FTPSRV user record, to appear as root when the user is logged in. Website uploads by third-party FTP client programs work as expected.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27943
19/10/2011 LPC.DRV Noticed a slight beat in stepper motors when running. Had noticed this earlier this year, but felt sure is was cured by running stepper program on core 2 of the Atom330. Definitely an interrupt of some kind, even though interrupts are disabled. Wrote a quick driver for the LPC, which clears Global SMI Enable. Job done, the interrupt is coming from ACPI timer overflow, every 2.34 seconds.
A side-effect of this is to route these interrupts to the processor, if desired. Added a quick interrupt handler, to catch power button press. Will expand on this in due course to provide a clean OS shutdown.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27943
18/10/2011 High speed timers - TSC Replaced very old time delay functions msecdelay,usecdelay, mseciowait etc with new functions based on processor Time Stamp Counter and 64-bit arithmetic. This obsoletes pre-Pentium machines for the time being, but simplifies the timers, increases accuracy and makes them independent of processor clock speed - so they generate the same time delays before and after a processor speed switch.
miniseconds( ) was the first function to use the TSC to schedule network events with sub-millisecond accuracy. Revised miniseconds( ) to use the same 64-bit arithmetic routines as the delay timers, so that time stamps and time delays in ChaOS now lie on a coherent time base.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27943
16/10/2011 Processor Power management Spent a day trying to overclock my i7-920, whilst monitoring the TM2 digital readout, eventually settling on a clock speed of 4.40GHz and minimimum voltage settings for CPU, CPU PLL and DRAM, instead of the BIOS [Auto] options. Makes sense really, because minimum voltage will generate minimum heat. Adding voltage to sustain a higher clock rate puts the Nehalem onto the ragged edge of automatic thermal throttling, or runaway into catastrophic thermal shutdown. It is easy to see this when watching the TM2 digital readout(DR) when the i7-920 is clocked around 4.5GHz+, 5-10 seconds at DR=0, and the processor switches off.
DR gives the thermal distance between current operating temperature and the factory-set maximum operating temperature, so I experimented with a few tweaks to see if this value can be increased (i.e. processor cooler) without hurting system performance in ChaOS.
As I write, with the i7-920 clocking 4.40GHz, DR has crept from 20 to 14, which is pretty hot. Simply adding a HLT instruction in the getkey() message loop causes DR immediately to jump to 34, which is a drop of 20 degrees Celsius! Even though the processor is restarted 100 times per second to service the timer interrupt, I guess it is suddenly 95% idle.
Added a system message counter, and integrated this into a message rate, recalculated every 1/100th second. When message rate drops to zero, I use the IA32_CLOCK_MODULATION MSR to throttle the processor to 1/8th speed. If the message rate rises I put the processor back to full throttle. This simple scheme works well; when idling DR increases to over 40, way over 25 degrees cooler..
All-in-all a very worthwhile exercise, a 67% on-demand speed increase from 2.67GHz to 4.40GHz, and a processor running cooler than before. Tried the exact same code on a Toshiba laptop, T4200 at 2GHz. Normally the internal fan on this machine kicks in about 20 seconds after ChaOS boots. Switching on this simple energy-saving mechanism calms everything down, heat belching from the left-hand vent reduces and the fan switches off. Mission accomplished.
For HT processors, using clock modulation control works only down to the fastest setting of the two threads on a processor core. Also, BIOS may leave SpeedStep disabled - this seems to be the case for Intel Atom N270 and D525 as far as I have seen. To ensure processor braking works from system start-up, I added a few lines to the AP startup module (used by CPU device to enumerate the APs), to make sure SpeedStep is on, then throttle back to 1/8th duty cycle before HLT. Thus the BSP will throttle back immediately on entering the idle message loop.
Care needs to be taken when changing processor speed during hardware access. The processor will sleep for a while and may miss a hardware interrupt.
Not yet decided how to handle speed stepping for the APs (which will start up at minimum speed under this scheme). First I will modify the system timers, to be smart enough to cope with dynamic changes in processor clock speed.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27966
11/10/2011 TCP/IP/HTPSRV Added code to scan HTTP headers for Hostname:, to see if I can host multiple websites on the ChaOS experimental server at 82.68.176.217. Added a copy of www.pedleyfamily.ws and mapped DNS for bob.ctpp.co.uk to the server. A baptism by fire really, as the home page loads a dozen or more links, which start arriving at the server even before the home page has been uploaded to the client.
Needless to say, my crappy little programs could not cope with the deluge, so I wrote a TCP browser, to scan for active connections, then to scan the ARP packet buffer for frames matching the selected TCB. This allows one session to be separated from the avalanche of packets hitting the server. Packet details, one per line with sub-microsecond arrival times show clearly what is happening.
Added a similar browser for the TCB tx queue, to examine the packets waiting to be sent, or to be acknowledged. Added keypress to cause a packet to be requeued. This is great fun, hitting the right packet causes stalled sessions to restart. Fairly easy then to add code to TCP/IP protocol driver to do this automatically.
In fact the main problem was not the TCP/IP, but the way in which TCBs were attached to HTTP sessions in HTPSRV. In a busy environment, TCBs are closed and recycled, and if identified as pointers can be confused with an earlier session. Fixed this by copying TCB->iss (start sequence number - which is a random number loosely advancing with time) into the HTTP session structure. There is then no confusion when messages from TCP are passed into HTPSRV.
Uploaded a revised HTPSRV.DRV to the server via ChaOS FTP, and rebooted the server remotely using a ChaOSNET UDP command, all running fine.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27932
11/10/2011 64-bit arithmetic on 32-bit processor On one of the backup DVDs I found a very old (1999) source directory with assembly language code for performing 32-bit arithmetic on the Intel 8086. What goes around comes around, so I have rewritten these routines, changing the old 16-bit register references to 32-bit and Hey! instant 64-bit support for 32-bit processors.
Although I am fast moving into native 64-bit programming, it will be really useful to have some 64-bit support in 32-bit ChaOS, not least because I need ways of inputting and displaying the 64-bit integers. Beginning to leave ChaOS v1.02 behind now, it is stable and good for the works invoicing and order processing etc, plus the ChaOS experimental servers. All the 64-bit stuff will be ChaOS v1.03 and higher.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27932
10/10/2011 ChaOS DVD+R, DVD+RW backups Have long since been using CDR to back up ChaOS, as well as test the ISO downloads. However all my early work is on FAT16 partitions, max 2Gb, a limit which I have used for many ChaOS development partitions since it allows 4 bootable non-EDD partitions in the MBR. It is a pain to extract subdirectories from a 2Gb drive to squeeze them on to a 700Mb CDR, so I decided to improve CDR1 and MC1, my device-independent optical media utilities to reliably create backup DVDs.
With this done, all of my current and historic partitions fit on to a single self-booting ChaOS DVD.
Had a go at DVD+RW, not a lot of people know that these disks need to be formatted before use, and this takes so long that the manufacturers have built a background format routine into the drives. Added code to MC1 to spot this, and start, or restart the format... with a please try later message.
Read performance from DVD+RW is a disappointment, so this is no substitute for a USB stick when moving ChaOS partitions from one machine to another, but once formatted these disks do allow random read-write access....
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27922
9/10/2011 ChaOS64 Added fRW flag to PCODE, which when set in the em64.drv PCODE table causes compiler back end to add a REX.W override to the instruction stream. I have used this trick for years to add OP and EA overrides to generate 32-bit instructions for the 16-bit real-mode code in the ChaOS bootstrap.
Added rRW to a selection of pcodes (ADD,SUB,MUL,IMUL,DIV,IDIV) and suddenly the expression analyser is generating 64-bit code for 64-bit calculations.
Added support for far function modifier, which has always been parsed by the ChaOS compiler, but never implemented (I have never had a need for it in the ChaOS flat 4Gb address space). This provides the means to switch from 32-bit mode into 64-bit mode by simply defining the entry function as far, and bracketing the function with #setsymbol pmode32 0x70.... .........#setsymbol pmode32 0x10
As I said yesterday callouts from 64-bit code to 32-bit routines will be a little harder to implement - but I forgot about the trick already used in ChaOS to implement calls by function pointer (ChaOS Diary 6/5/2009). By placing a far pointer on the stack, a far call can be implemented using call far [esp+00]. Finding the right target segment should be just a question of pushing the right value on the stack at run-time. Thus far calls to transfer control between 32-bit and 64-bit segments would be implemented quite painlessly, once again without the need for call gates.
Update: This method works fine to transfer control from 32-bit to a 64-bit function, and vice versa..
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27922
8/10/2011 ChaOS64 64-bit pcodes and far call mechanism ModRegR/M 05 opcodes in 64-bit native mode generate RIP-relative memory addresses, and with the new 64-bit development environment up and running I can for the first time see this happening. By Sods Law these opcodes are used extensively in ChaOS, but having created a mechanism in em64.drv to substitute the PCODE table, I can use ModRegRM/SIB 04/25 opcodes instead when em64.drv is running. These opcodes work fine in 32-bit mode also, so em64.drv PCODE table can be used to compile 32-bit code. Just recompiled the whole OS using these PCODES, runs fine (but each of the affected instructions adds a byte to the length of the code segment).
Near and far call via 0x9a opcodes are invalid in 64-bit mode, so there is no way of encoding a direct far call out of a 64-bit segment. Control transfers from 64-bit to 32-bit (e.g. debug handlers) must be via an indirect memory address. The most painless way to glue this into the 64-bit compiler is to require the use of a far pointer to function to invoke these rare types.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27910
7/10/2011 ChaOS64 and another function modifier: hook Existing hook mechanism works on a symbol name match only, and replaces references in just one OS process. Decided to add yet another function modifier so that hooks can be detected in the same way as dynamic links, as a program loads.
A ChaOS hook is like a dynamic-link in reverse, rerouting all existing calls to a given function to the newly-loaded version. By embedding this feature within the program load code, and using the same symbol-matching regime as a ChaOS dynamic link, hooks become type-safe links, and thus much more useful.
Made a few changes to the inbuilt compiler back-end, so that the PCODE table entries are found via a function, rather than by direct array references. hooking these functions allows em64.drv to substitute a larger PCODE table, which I am using to add 64-bit native features. The existing compiler will then generate 64-bit instructions when em64.drv is loaded.
Also added unhook function, called on hot reboot to scan for OS function addresses which map outside the OS code segment; same as above, the functions can be remapped, and the OS can restart without any drama.
This will speed up the development cycle considerably, as the two main tools for generating and testing 64-bit code (i.e. the compiler PCODES and the disassembler) are now contained in a separate, loadable module. ChaOS falls back to regular 32-bit-only behaviour if this module is deleted.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27906
3/10/2011 ChaOS64 After such rapid progress with mutex and offload mechanisms, decided to advance 64-bit development using ChaOS v1.03 as a platform, rather than destabilising v1.02
Fortunately, the ChaOS compiler model is extremely simple, and uses only two processor registers (EAX and ECX) most of the time. This has left EDX free which I now realise is ideal for carrying the upper 32 bits of 64-bit temporaries in the expression analyser.
Very quickly added support for 64-bit unsigned integers, via strtoul64 function, modifications to doinitialiser,fetch,store, getasictype,npushargs, along with 9 extra pcodes to generate processor instructions to load and store the high dword of 64-bit operands from EDX.
Added a new protocol server em64.drv, as a container for 64-bit enhancements. As a start, em64 hooks disassemble, providing better disassembly of 64-bit instructions, REXW, etc.
Next will be a new compiler directive to set pmode64, which will cause the generation of a 64-bit code stream. I will use em64.drv as a container for further hooks, as necessary to generate this code. This keeps 64-bit development in one place. If em64.drv is missing ChaOS will fall back to normal operation.
* direct diary upload via FTP1 (PUT command) from 4Gb CFS SATA partition sab1: over ChaOS version 1.03.27842