ChaOS Home ChaOS Source Notes ChaOS Source Index ChaOS Downloads CTPP Home
Current Diary Diary to Mar 2010 Diary to Dec 2009 Diary to Aug 2009 Diary to Apr 2009 Diary to Nov 2008
ChaOS Diary - monoblog and links to reference documents
Many golden nuggets lie herein.
30/6/2010 ChaOS IP protocol suite Added STOR to ChaOS FTPSRV at chaos.ctpp.co.uk, to allow direct uploads to the experimental servers. Managed a couple of uploads from ChaOS box via ChaOS FTP client but did not work every time. Leaving the server for a while seems to clear the problem. Drag and drop into a Windows XP ftp://chaos.ctpp.co.uk session works fine. This suggest the server code is OK, just need to refine the ChaOS FTP client.
* diary upload direct from CFS development drive
29/6/2010 chaos.ctpp.co.uk Broadband connection Little change with the new telecoms, we are too far from the exchange. I can manage a steady download from chaos.ctpp.co.uk around 68kbytes/sec, which appears to be above the quoted router uplink speed of 448kbit/sec. Maybe the ADSL uplink uses compression to achieve this packet rate.
* diary upload direct from CFS development drive
28/6/2010 ChaOS IP protocol suite Small improvements, all positive. Running chaos.ctpp.co.uk on a 400Kbit uplink, with 8 IP addresses sharing the same line, is a severe test. Uploading packets to the internet with intervals below 8ms just doesn't work, this could be due to the surveillance cameras streaming at the same time on two of the IP addresses. Zen will be installing new equipment at the exchange tonight, so tomorrow will show whether the speed problem is on the uplink.
Added code to read DNS MX records and find mail servers associated with a given hostname, this in preparation for a simple SMTP server for ChaOS. Wrote quick program to do this for a hostname supplied by the user, followed by a TCP activeopen on the mail server. Amazed at how long the mail servers take to respond to a SYN, up to 15 seconds, good ammunition for writing a server of my own.
Tried the same code at home, doesn't work - TalkTalk Broadband name server returns only the mail server host name, not the A record, so a recursive DNS search is needed. Zen Broadband name server at work returns all the mail server A records as Additional RRs.
* diary upload direct from CFS development drive
27/6/2010 ChaOS IP protocol suite Steady progress on new IP protocol suite (TELSRV/HTPSRV/FTPSRV/FTP/HTP/TELNET). The chaos.ctpp.co.uk experimental server is still very slow, even though the same code over the LAN or through a faster broadband connection works fast and furious. I know there is little wrong with the TCP code now, so will try small changes (e.g. increase cwnd to two segments for TCP slow start, add a delay between those first two data segments).
Rewriting the TCP broke just about everything in the ChaOS IP protocol suite, so it will be a while before functionality returns to that achieved with the old TCP. I will follow my usual method of implementation on an as-need basis. Having said that, the new TCP is less than 10% of the size of the old, with far greater potential.
Added SYN timout, to abandon session if three-way handshake not exchanged within 5 seconds.
* diary upload direct from CFS development drive
19/6/2010 TCP/HTPSRV/FTPSRV/TELSRV 5 minute inactivity timeout added to TCP for all sessions. ARP cache increased to 200 entries.
* diary upload direct from CFS development drive
18/6/2010 TCP/HTPSRV/FTPSRV/TELSRV New TCP has now been running for 4 days at chaos.ctpp.co.uk, and is clearly more stable than before. The TCB table is filling up slowly with sessions which have ended abnormally. I will add a timer to send RST|ACK after 5 minutes of inactivity on any open TCB, irregardless of the session state.
* diary upload direct from CFS development drive
14/6/2010 TCP/HTPSRV/FTPSRV/TELSRV Second embodiment of the new TCP, HTPSRV HTTP header fixed, ported TELSRV and FTPSRV, all three servers now running at chaos.ctpp.co.uk. TCP bulk send is too slow, because slow-start and fast retransmit/recovery are not working correctly yet.
* diary upload direct from CFS development drive
13/6/2010 TCP/HTPSRV/FTP/TELNET Installed new TCP and HTPSRV at chaos.ctpp.co.uk, but HTTP header returned by the server over the internet is malformed. Not a big problem but this did not happen in testing over the LAN. Ported ChaOS FTP to the new TCP, no major issues, and used the new FTP to upload this diary update. Ported ChaOS TELNET to the new TCP, again no major issues.
* diary upload direct from CFS development drive
11/6/2010 TCP/HTPSRV Back to rfc793 to nit-pick TCP state transitions, realised that the user close() function behaves differently according to which end of a connection send the first FIN. Added tcpREMOTECLOSE notification after transition to CLOSEWAIT, to signal to user that a call to close() is expected. Added a switch in close() to handle ESTABLISHED and CLOSEWAIT case. With this in place, the new ChaOS TCP state machine is functioning broadly as rfc793 page 23.
* diary upload direct from CFS development drive
9/6/2010 TCP/HTPSRV Added user notification mechanism to new TCP. Revised new HTPSRV to use tcpNEWSESSION notification to open HTTP session. At this point I also call passiveopen() to create another LISTEN port for the next incoming SYN. This keeps server-side TCP sessions to a minimum, only one LISTEN session is needed for each idle server.
Added TCP ctl parameter to send(), flags which will be sent on the last TCP segment. To preserving a session after the send completes, tcpPSH can be passed to indicate end of HTTP file transfer. To end a session as the send completes, tcpFIN can be passed to close the PASV data session of an FTP file transfer.
* diary upload direct from CFS development drive
7/6/2010 TCP/HTPSRV HTPSRV ported to new TCP, working well enough to try hosting experimental server at chaos.ctpp.co.uk tomorrow.
* diary upload direct from CFS development drive
6/6/2010 TCP New TCP now functional, with activeopen, passiveopen, send, receive and close working. Basic TCP/IP sessions established with Zen FTP server, XP Telnet, HP JetDirect, FireFox and Explorer.
* diary upload direct from CFS development drive
5/6/2010 TCP Decided to tackle complete rewrite of ChaOS TCP layer, adhering more closely to rfc793, but using auto-resizing buffers for send and receive, and a similar transmission queue. This is an idea I used for my print spooler PRNSRV, which works well enough. When ther is insufficient space in a buffer for incoming data, I double the buffer size, add the incoming data length and realloc a new buffer. Data in the buffer is always accessed through bufused and bufptr which are offsets from the buffer base address, not memory pointers. Other functions using the buffer always read the latest buffer address to find data in the buffer, and are unaffected if the buffer is resized and moved to a new memory location. This method means only one resizable buffer is needed for each TCP session. Outgoing segments copy data from buf+seqno-(iss+1), and packets which are retransmitted can use the same calculation even if the buffer has been resized in the meantime. This is much neater than allocating small buffers for each outgoing segment, on the slim chance that the segment needs to be sent again. A similar resizing scheme is used for the transmission queue.
Already behaving better than my first attempt at TCP, this design will gather statistics for every outgoing segment in a session, particularly transmission and ACK times. This real-time analysis will assist the development of TCP throttle controls, which need RTT (round trip time) calculations.
* diary upload direct from CFS development drive
3/6/2010 FTP Just noticed some downloads by ChaOS FTP are truncated. This is because I save the data channel on receipt of the FTP 226 data transfer complete message. Now only an American would say a data transfer is done when in fact the packets have not yet left the server, but, who am I? I only know I receive the 226 before the data has all been received. This seems to happen only when the file transferred fits into two IP packets (TCP slow-start?). At the moment the fileinfo for the data transfer is held in the FTP control structure, which cannot know when the PASV TCB is finishing. I can handle this condition by breaking into the debugger when this happens. After a short delay the data channel completes and all is well. Ultimately the job of downloading the received file needs to be delegated to the data TCB on receipt of FIN.
2/6/2010 Diary on CFS Diary master copy transferred to CFS development drive. This diary entry was added using ED (the ChaOS source file editor), followed by a direct FTP upload to the Zen server using TYPE I,PASV and STOR.
1/6/2010 High Speed timer Revised miniseconds() function to cope with lost timer interrupts, which occur during real-mode callbacks. Uses processor time stamp counter to estimate number of interrupts lost and the elapsed time in nanoseconds since the last 100Hz timer interrupt. Now produces reliable times for network packets which can be used for TCP/IP scheduling with accuracy down to a few microseconds. Yes I know there is a more reliable timer on modern processors, but my older computers have no APIC, so this method works for them by intercepting the invalid opcode exception caused by RDTSC, and setting EAX and EDX to zero which causes the high-speed timer code to fall back to 100Hz granularity. At least this way I can use the same timer code on the widest range of processors. miniseconds() would return slightly inaccurate nanosecond times if the processor clock is changed, but adjusts automatically to the new speed after two consecutive timer interrupts.
25/5/2010 CMPP/Batch file arg substitution Revised CMPP (compare project) to compare project make files before comparing source files. Later version overwrites make file on alternate drive, and lnkext(LDRIVE*) ensures the correct file extension (.lnk or .link) is used according to logical drive attribute, maintaining the back-link to FAT16. Added optional arg2 to CMPP to override project name, so projects can be compared without changing the current project. If arg2 is used, auto flag is engaged and all files are synched without asking for overwrite permission. Added /i option to CMPP to compare and update header files between in the /i directory.
Added %1 through %9 arg substitution to shell batch file handling. A small batch file using a variable arg for logical drive can now synchronise several projects with that drive, even if it uses a different file extension set.
23/5/2010 ChaOS Source Index/FTP upload/FTP download Revised ctpp.co.uk ChaOS Source Index, replacing all old file extensions with the new lower-case .html and .link. Added Redirects in .htaccess to map the old filenames to the new where appropriate. Revised and improved FTP (which can accept simple script files, using .ftp file extension), and increased MAXTCBS in IP4/TCP to 200 sessions, to mirror the whole website back to a ChaOS partition. Improved FTP upload, and added upload.ftp script file to each ChaOS project currently represented on the website. Added ide driver to website, just to check that directories can be created by an extra line in upload.ftp. I need to write a simple HTML editor maintain this whole website from my ChaOS box, long-term this will be much easier than clunking around .cPanel, as the FTP MDTM command allows me to check which files are out of date.
23/5/2010 CFS File extensions CFS development partition now stable enough to add another file extension to extset 1 (.lnk-->.link), switched on by system id 0xce in partition table. Backed up CFS development partition, synch with old FAT16 development partition, then create a new CFS partition, copytree FAT16 --> CFS to produce new CFS partition with all .lnk files now appearing a .link files. System now ready to revise website source index, knowing that file extensions are now unlikely to change.
22/5/2010 A Simple ChaOS router/packet sniffer ChaOS ARP driver has always allowed for multiple network cards, and if these are all connected to a DHCP router they will grab individual IP addresses. Had an idea that a ChaOS box with two network cards might be able to act as a network switch by operating a duplex packet relay, retransmitting packets received to the alternate network. Since all packets are buffered in the ARP driver, this would provide a means to examine the packet traffic on any ethernet connection, to better understand how to implement the more complex protocols such as HTTPS.
Using a simple scenario: DHCPDISCOVER broadcast received by network card on ChaOS box AND receiving network card has no IP address (not connected to DHCP router) AND alternate network card exists in the system with an IP address (connected to DHCP router), a packet relay can easily be implemented to act as a software switch between the two networks.
After some experiment, the key to this method is to program both network cards in the bridge to receive all packets with physical address, (usually bit 0 in Receive Configuration Register), so they respond to frames with any MAC address, not just broadcasts and exact MAC address matches. On the transmitter side, nothing needs to be done, as the hardware will place any source MAC address on the wire. By restricting the relay to a particular MAC <-> router MAC combination, this relay works without disrupting other local traffic.
Using the DHCPDISCOVER broadcast as a trigger to switch on this packet relay (rather than messing about with routing tables) confuses Win XP for a short while on boot up or after a resume from sleep, but when the DHCP is refreshed all network sessions*, including HTTPS run OK.
*connecting direct to the router MAC address will be a special case
16/5/2010 CFS/FTPSRV/HTPSRV/TELSRV A baptism by fire now for CFS, trying to run all the ChaOS server programs along with the compilers over this new filesystem, whilst maintaining the backlinks to the FAT16 partitions through the file extension translation tables. Issues resolved today centre around virtual subdirectories, and making sure these are not returned to functions which expect unique filespecs, e.g. findfile(), uploadfile(), downloadfile(). So far FTPSRV, HTPSRV and TELSRV servers perform identically over CFS or FAT16, which is encouraging. Will try a CFS partition on the chaos.ctpp.co.uk server this week to see how it copes with moderate internet traffic.
A few tweaks have already been added to CFS to save and restore the binary index context where file search functions are nested. All filesystems need a FILE structure in which to carry the filesystem context for findfirst/findnext sequences. At present only a CHAOSFILE structure is used to carry this information, which is a throwback to the original CFS design. With the addition of a binary index, the context becomes the current binary index pointer, and end-record key. In essence, a new DBSHELL structure could be created by findfirst(), which can then be used to find files in the master binary index, using a local search context. Leaving aside the potential for files being writted or deleted after the local DBSHELL has been created, this allows multiple findfirst/findnext sequences to operated concurrently.
Handling dynamic changes is more of a challenge. The master filesystem may need to keep track of DBSHELLs in use, and apply the changes necessary to keep them up to date. Extra discipline may be required to notify the master filesystem that a DBSHELL is no longer needed. Or local DBSHELLs might be disconnected after a short timeout (or a binary index update), blocking findnext() calls for outdated DBSHELL contexts.
A more pragmatic solution may be to increment a counter each time the master index is altered. If this is copied into a local DBSHELL by findfirst(), then findnext() can immediately know the filesystem has changed by comparing this value to the master index counter. Instead of a simple READNEXT(), the local DBSHELL needs to be updated with the new index parameters, and update the endrec key (the filesystem change may have altered the subdirectory branch index table, etc). READ() on the new index will reposition the local DBSHELL binary index pointer on the correct file record returned, READNEXT() will then advance to the next file in the new binary key order. START(KEYGE) followed by READNEXT() will allow the local DBSHELL to continue a findnext() loop in the rare event that the current file has been deleted!
More sophisticated filesystem behaviour, such as notifications to window processes displaying file listings could be implemented by registering index ranges with the master filesystem. Since all directory listings in ChaOS are in binary key order, the start and end of any directory (with or without its subdirectories) is represented by two integers. These values can be compared to the index range affected by a filesystem change, and a notification message issued as necessary. Notifications should only be needed where the update region overlaps the registered region, because the displayed files would not change even though their index positions have been shifted. Any refresh of the file listing should of course check the master index counter, and generate a new DBSHELL for the display update.
16/5/2010 FTPSRV/Mozilla Firefox Tried sending a directory listing (as for FTP LIST command) in response to Mozilla Firefox RETR <directory> request, but with no success, Firefox seems to be expecting something different. Replying with a code 550 causes Firefox to issue CWD and LIST, which can be handled normally.
15/5/2010 CFS/FTPSRV Now working fine for Win XP FTP client sessions, just needs a tweak for Mozilla Firefox which uses RETR <directory> rather than CWD <directory>, LIST when changing directories.
11/5/2010 CFS makevirtualdirectory() added to return PATH structure with ftSUBDIR attribute for each file set found with nesting level one greater than the current directory. Although CFS has no physical directory structures either in memory or on the disk, when copying file sets from drive to drive, or passing file information through FTP server, other operating systems and their filesystems need to see a virtual directory in order to navigate the CFS binary index.
3/5/1020 CMPP Compare project now modified to correctly compare projects across filesystems with different file extension sets. Due to the difference in file time granularity, when comparing CFS to FAT16, the CFS file time is now rounded down to the nearest even second (FAT16 time is only accurate to 2 seconds) before comparison to the FAT16 time. This ensures equivalance of file time after a file synch between different filesystems.
3/5/2010 CFS Boot CD Added a few mods to MC (Make CD) to allow CFS partitions to be archived as a bootable CD or DVD, using the native CFS bootstrap. Special attention given to make sure CFS partition is saved with the CFDBASE, for fast loading of the binary file index, this exposed a slight error in padding the last iso sector out to 2048 bytes, now fixed. When making a more sophisticated boot CD, with a full set of device drivers, a new bug surfaced when copying wildcard file sets from one CFS partition to another (this is a new thing!). Both the wildcard search, and CFS file upload use the DBSHELL binary end key - so this is fixed by adding code to CFS file upload and download. DBSHELL state is saved before using the binary index, then restored afterwards.
3/5/2010 CFS File extensions After a little more thought, it is more logical to control development file extensions through multiple tables and a logical drive attribute, rather than just one fixed operating system table. This allows file different file extension sets to be used by the same compiler, switched according to the drive attribute of the current project. Better still, CMPP (compare project) needs only a slight modification to compare projects on different drives with different extension sets, and correctly translate extensions when synchronising source files between the two. Most importantly this allows CFS with new file extensions to be run as the lead development system, without losing the ability to quickly transfer a project back to compile over FAT16. Similarly file extensions translated during a drive->drive copytree will be translated back if the copytree is in the reverse direction.
1/5/2010 CFS ADV CFS now stable enough to run with .html source file extensions throughout, and to boot by default, but coexisting with the FAT16 system throws up a number of issues. Most issues relate to my goal to revise file extensions. It is easy enough to filter file extensions during copytree, to create a CFS partition with a revised file extension set. But for other commands such as file copy and file compare, the file extension translations need to be reversible, and all logical drives need to be flagged for the file extension set in use. Then a .html file copied from CFS to FAT16 will automatically arrive as a .htm file. Similarly .htm filenames recorded in the editor control files on FAT16 need to be silently translated to .html where the project is copied to a CFS partition.
30/4/2010 CFS ADV Improved CFS dir command, to display subdirectories. Because there are no physical directories on a CFS disk, subdir flag passed to findfirst causes SETENDKEY to include subdirectories in the binary search results. Files in subdirectories will thus be returned to the findfirst/findnext loop. Because the binary index returns the files in exact alphabetical order, all that is required is to display the next lower tree branch each time it changes.
26/4/2010 SLPSRV Started a small server project, to handler Service Location Protocol as used by the JetDirect interface. This server needs to load after the network ARP and TCP/IP protocols, and after the device scan (a network card is essential!) but before the print spooler PRNSRV. At the moment TCP/IP is level 3, ARP is level 2, then the device drivers load, and PRNSRV then loads with the rest of the message servers at level 1.
Revised the server levels, to make space for SLPSRV; TCP/IP is now level 5, ARP is level 4. Device drivers load, then SLPSRV will be level 3, TELSRV,FTPSRV,HTPSRV are level 2 and PRNSRV is level 1.
SLP is a subset of UDP, and at present ARP processes a limited number of UDP protocols. As a quick fix, I have added a small module slp.htm to ARP, to catch UDP packets on port 427, and store attribute frames in a table which can be scanned later for device info, leaving the SLPSRV project for a rainy day.
25/4/2010 SLP/Multicast/HP JetDirect Opened a jar of worms trying to autodetect HP JetDirect printer device on LAN. First of all I had no idea what multicast addresses were about, eventually I found the JetDirect interface responds to IGMPv2 type 0x11 all routers multicast on 224.0.0.1 with type 0x16 reply to the hp-discovery multicast address 224.0.1.60. This packet carries the IP address of the JetDirect card. This is only half the solution to the job of autodetecting HP printers on the LAN. I have yet to construct the correct packet to trigger the printer to send out the SLP attribute packet, which carries the printer model, ID, language, IP and MAC address.
There is a simple Telnet server in the JetDirect card, which allows manual configuration of most interface settings, though some commands are undocumented. The hidden command ttl-slp:1 causes the interface to send out, three identical SLP attribute packets to the multicast address 224.0.1.60:427 during the power-up sequence, and these are easy to catch provided a simple SLP server is already up and running.
Update: Writing the above note caused me to wonder whether Telnet can be used to trigger the SLP attribute packet, and the answer is yes it can. ttl-slp:-1 turns off the SLP multicast, and quit saves the settings. Then start a new Telnet session, send ttl-slp:1, then quit, and the JetDirect responds a couple of seconds later with one SLP attribute multicast. I knew there had to be a way!
23/4/2010 HP JetDirect Having succeeded in sending print jobs to my HP LaserJet 2200/2300 printers via USB, decided to try the HP JetDirect interface. After a couple of hours I realise JetDirect contains a simple FTP server. FTP command sequence PASV,TYPE I,STOR,QUIT are all that are needed to upload a job to the printer.
21/4/2010 HTPSRV/TELSRV/FTPSRV Machine running experimental ChaOS servers on 82.68.176.217 chaos.ctpp.co.uk has been locking up from time to time since going live last December. Latest ChaOS version 1.02.26714 seems to be more stable.
21/4/2010 USBPRN Tried NJOB over PRNSRV over USBPRN to load HP Laserjet macros for order and invoice print, plus report printing. Works fine. As with all new device support, the hardest job is getting the first simple driver to work. Much time is spent fathoming the hardware, often by trial and error, before the device springs into life.
20/4/2010 USBPRN Dusted off USB code to write a simple bulkout printer driver for dev/class/protocol 0x070103. Redirected output from PRNSRV to PRN device on USB, when parallel port driver is not found.
19/4/2010 CFS ADV Slight error in code used to combine adjacent free blocks during chaosfree(), have been testing with this feature disabled but of course this means that wildcard file delete leaves lots of small free blocks in the filesystem, not how I want things to be. The reason is the use of multiple empty.item files to hold the free blocks, which requires a findfirst(empty.item), then subtract 1 to get a free archive number to create a unique binary key for the new empty.item record. I had realised that findfirst(empty.item) repositions the binary index, which would disrupt the outer findfirst()-findnext() sequence of a wildcard delete, but saving and restoring the binary index position is only half the answer - when chaosfree() combines adjacent free blocks, the index shrinks by one record; at this point, if the current binary index position is above the record being deleted, it too must be decreased by 1, otherwise a wildcard search may skip a record.
The result is as intended - when there is only one large free block in the filesystem, I can copy a whole bunch of files on to CFS, and they will be written to adjacent blocks on the disk. If I then delete them one-by-one, or in groups, various free blocks arise in the filesystem, according to the physical sectors used. But when all the files in the bunch have been deleted, the space freed coallesces into one free block again.
17/4/2010 CFS BOOT File extension definitions for compiler, linker, and project make now collated in intcmd.htm, with the shell command set. This makes it possible to redefine file extensions for the whole operating system without breaking the compiler. Added a file extension filter to copytree to convert file extensions during a drive to drive copy. Used this to create a CFS bootable drive, with all .htm extensions changed to .html, then recompiled OS1 with htmext also set to .html. Booted this version of OS1 on the CFS bootable drive, to create a version of ChaOS which now uses .html for all source files. Everything working fine. The same method can be used to modify other critical file extensions such as .job and .lnk which clash with Microsoft filenames.
16/4/2010 PRNSRV/1284 PCI Added a simple driver to detect PCI parallel part adapter and store primary port at *(UL*)0x408, which makes PRNSRV detect the adapter and treat it as LPT1.
14/4/2010 PRNSRV/COB Serendipity day. A bit of code I use to drive a parallel port malfunctioned when embedded in PRNSRV, when trying to load macros and print to a LaserJet printer from an old Pentium II machine. Tried using the printer BIOS function and this works fine, so traced the BIOS code, and found an out(0xeb,n) instruction after each write to the parallel printer ports. Not knowing what this is, I assume it is either a time delay, or a port which needs to be written to assert a port write on to the ISA bridge. Whatever, adding out(0xeb,n) after each port write in PRNSRV fixes the problem and my documents print correctly.
Have also been experiencing trouble with the 8255 I/O card which drives the Cobden Chadwick print machine. This is an even older system, a P75, and the relays driven by the 8255 have been misbehaving, especially in the extreme cold winter we have had. Things settle down if the system is left powered up for a couple of hours, so I had always put this down to damp in the works. Tried adding the outp(0xeb,n) to each output in COB, the version of ChaOS which drives this system. Everything worked fine today, so maybe this is the answer to handling 8255s on older systems, time will tell.
12/4/2010 CFS ADV One last tweak to CFS binary indexing, reserve memtree branch index 0 for root directory, by adding 1 to any index read from treexref[]. Directory nesting (depth) count can be eliminated from the binary index key whilst still discriminating between root directory files, and files in subdirectory with temporal name index zero. Initially this error did not show, because there are no files in my main project subdirectory which occupies temporal name index zero, the project files are in the second directory branch nesting level.
11/4/2010 CFS BOOT CFS device driver mechanism now working. Rebuild OS kernel over CFS is much faster than FAT16, as expected, 10 seconds instead of 16 seconds. Reboot mechanisms all working fine, bearing in mind CFS uses a slightly different bootstrap.
10/4/2010 CFS ADV Because the binary indexes are so quick and easy to build, I added a consistency check after every file creation event, comparing the main CFS index to a new one. This now proves CFS is working as designed. If I ever need advanced filesystem features such as search by file modification time, search for a given filename, an alternate binary index can be created with different key start and key length to view the filesystem in a different order. This takes a few milliseconds, rather than the minutes seen when searching NTFS for a particular file.
Added CFSYS command, along with hacked-down bootstrap to provide a boot mechanism for the CFS ADV filesystem. Not bothering with CHS addressing, the bootstrap can be loaded in toto by one EDD packet read command, meaning the CFS boot sector is more than half empty. LBA and length of the bootstrap are all that is needed, along with LBA and length of the OS kernel image, these numbers are written into the boot sector as relative LBAs from the boot sector itself. Just booted successfully for the first time from a CFS partition, next needed is a revised bbd and drv mechanism to load the device drivers, and I can retire FAT16.
8/4/2010 CFS ADV New CFS with operational binary index now working in prototype. To preserve the existing CFS, the binary indexes are switched on by a single flag. Copytree (XCOPY) of 5400 files from FAT16 to CFS ADV takes 86 seconds, which is the same time as copying FAT16 to FAT16. This prototype updates the subdirectory branch index every time a new subdirectory is created, effectively keeping the whole filesystem, in perfect alphabetical order after every file creation or deletion. Because the index is entirely in memory, the time taken to keep things in order is not measurable. This is unlike MS windows and Linux extFS, which have to sort file lists to display them in alphabetical order. I have reworked my old archive system, which keeps backup copies of all files rather than overwriting them, but using negative numbers for the archive number. By starting with an archive number of -1, adding -1 each time a file is archived, this ensures the latest copy of a file appears first in binary key order.
The big difference I see is in the time taken to scan source files, check dependencies and compare file dates/times, in order to decide which modules need to be recompiled to rebuild a project. For a 120-module project the compiler the load time drops from 5.36 seconds to 1.2 seconds, for the linker the load time drops from 6.0 seconds to 2.2 seconds.
So now I have my dream filesystem, automatic archiving, no directory sectors or file allocation tables, no fragmented files, and no need to ever sort filenames to get alphabetical order.
5/4/2010 CFS Binary index code now added to ChaOS kernel, with one extra structure to solve the directory branch conundrum. I stated a couple of days ago that directory branches can be placed in alphabetical order by using another small binary index, but this is only true if the directory branches are saved in a proper indexed database. At the moment my database design serves only fixed-length records, so the packed CFS nametable would need to be expanded into a large table. At first glance, I thought the current branch indexes could be used as an index into the database position table, but in fact the extra structure needed is a reverse cross-reference table, which then successfully converts temporal indexes to alphabetic order. Simple, not really.
3/4/2010 CFS Completely overhauled CFS today. Disk layout is the same as the 2000 CFS, a chain of file markers very much like a memory heap, with the markers cached in memory for faster access.The simplistic disk layout for CFS is chosen to maximise the odds of recovering from a hardware or software failure. First of all there are no directories or cluster tables, and secondly files are never fragmented. Hard experience has shown me that most data loss occurs when directories or FATS become unreadable, and fragmented files are the hardest to recover successfully. Instead, all files on a CFS disk lies on a doubly-linked list of node sectors, each marked with a signature. The chain can only be broken if software or hardware causes the loss of a node. Statistically the bad sector is likely to be within a file data area, but if not, the chain can be read in forward and reverse directions to determine the location of one missing node. Failing this, the disk can be scanned for the node signatures before and after the unusable area, and one new node can cause the bad sectors to be avoided. To minimise the possiblilty of phantom signature nodes confusing a future recovery program, unused nodes are always written back to disk without the signature when thay are discarded. A simple switch causes files of the same name to be archived, rather than overwritten, and this causes disk wear to be spread steadily across the disk surface, whilst holding multiple backups of the busiest files in the system.
So the layout is inherently reliable, but suffers from a linear search time penalty. The first step in reducing the file search time is to cache the nodes into memory. I did this back in 2000, but had less memory in my machines then, so I never fully debugged the code, and fell back on FAT16. Today I have finally got the CFS cache working properly, and set to work on the performance enhancements. Revised and improved the code for file deletions, to check for adjacent free blocks above and below, and coallesce these into one free block. Free blocks within the main file area are now never subdivided when storing a new file which fits into the block. This results in a little lost space, but still far less than using a cluster allocation strategy, and prevents the accumulation of tiny holes in the heap. If there is no hole in the heap large enough to store a given file, the main free block at the end is divided into two nodes.
With the cache running reliably, I now store a backup copy after a short inactivity timeout (I use 2 seconds). The large heap free block is the obvious place, and this can be used to avoid the time delay in reading the nodes up from disk as the system starts. Typically 2500 nodes/sec becomes 125000 nodes/sec. I will do this when the startup delay becomes boring, but at present it is only 2 seconds for the 5000 files on my development partition.
Also reorganised the node structure today, replacing little-endian integers with big-endian ones, are arranging the file path information in better key order. The node cache is now a simple fixed-length record database, and can be indexed using the NJOB/DBASE binary index code. A quick prototype to see how this may pan out shows I am on the right track. 5000 files, 0.04 seconds to load the node cache backup from disk, and the binary index takes 0.004 seconds to build, allowing display of the whole filesystem grouped into directories, with files appearing in alphabetical order. So it is possible to provide directory-style access to files on a disk without using file allocation tables, inodes and directory sectors.
The final part of the jigsaw is to index the subdirectory name segments, which at present appear in order of creation. Of course a small binary index table for subdirectory names resolves this.
2/4/2010 CFS Performance testing the CFS file system today, with mixed results. Testing requires that large numbers of files are transferred on to a CFS partition, but generic filesystem calls such as mkdir(), isdirectory() have to be handled even though CFS has no directories. To fake a result for isdirectory(), CFS needs to scan all filesystem records for a match, which is quite slow at present, even though the subsequent mkdir() is irrelevant. Returning 1 from chaosisdirectory() with no file record scan allows a reasonable speed comparison to be made between CFS and FAT16, typically CFS finishes in 60% time. But this breaks much of the code used to parse command line file arguments.
2/4/2010 D10 over ChaOS Just checked the roller left running last night. Roller engraved OK.
1/4/2010 D10 over ChaOS Final tweaks, then run CO2 laser to engrave first full roller using D10 running over ChaOS operating system. The 188% dispro reported a couple of days ago was partly a red herring - whilst there was a compiler error in the casting of constants, this only resulted in a stray value being left on the co-processor stack, which did not affect subsequent calculations. And whilst a small test burn ran successfully, a full-width burn still exhibited the dispro error in the x-direction.
Sorted the problem today. About 5 years ago, when adding code to provide continuously-variable dispro scaling in the X-direction, I added a small code section to cope with arithmetic overflow when calculating rxloops. Obviously, to provide dispro scaling over the width of the whole roller, a Bresenhams loop needs to know the number of timing loops executed during the course of a complete engrave cycle, and 625KHz times several hours generates numbers in the region of 10,000,000,000 to 20,000,000,000 which will not squeeze into the 32-bit integers used by the laser timing loops. To avoid overflow, dbrxloops first calculates the really big numbers as floating point values, then divides dbrxloops and dbrxsteps by 2 repeatedly until less than 4294967296, i.e until the result fits into a 32-bit unsigned integer.
This has worked fine until this week. First problem is that the ChaOS compiler mistakenly compiles 4294967296.0 as 1.0 - a throwback to the fact that the code used to parse large constants was written before I added floating support to ChaOS. This can be easily resolved by using floating point code to parse the numbers, but this approach may not be ideal, because the bootstrapped ChaOS compiler will then always require a co-processor to compile floating point code. I looked at a Sun Microsystems source code file for the strtod() function, and was amazed to see floating point constants in there - this is fine if you already have a compiler which does floating point, but no good as a strtod() function in an integer compiler used to create the floating point code in the first place. This is job for another day, but with a little careful effort, I can ensure the ChaOS compiler uses no floating point instructions to create a floating point program.
Second problem is that the FISTP integer store instruction used by ChaOS treats 32-bit memory operands as signed - there is no way to do a 32-bit unsigned store from the FP. So even though rxloops is declared as a UL (unsigned long int), the assignment rxloops=dbrxloops stored a signed value - and if this is more than 2147486347 the FP will store the indeterminate value 0x80000000 (or raise an FPU exception if this is enabled).
So this was the actual cause of the dispro problem, numeric overflow in one of the seed values for the X-axis Bresenhams loop.
In the course of isolating this error, I have noticed that it is possible to load and store 64-bit integers to and from the FPU. Not sure which version of the Intel processors added this feature, but it is going to be extremely useful - it provides a really easy way of adding 64-bit integer support to a 32-bit system. Just add inbuilt 64-bit integers to the compiler, and code in fetch() to load these values on the to FP, and in store() to offload them from the FPU stack top. Although I have just declared it is a bad idea to make the ChaOS compiler dependent on the FPU, this provides a cheap and cheerful way of adding native 64-bit integer types, in preparation for a 64-bit version of ChaOS.