ChaOS Home ChaOS Source Notes ChaOS Source Index ChaOS Downloads CTPP Home
Diary to Jun 2010 Diary to Mar 2010 Diary to Dec 2009 Diary to Aug 2009 Diary to Apr 2009 Diary to Nov 2008
ChaOS Diary - monoblog and links to reference documents
Many golden nuggets lie herein. Trademarks are acknowledged as copyright of their respective owners
29/7/2010 Realtek Multicast Programming Tried a few times in the past to understand Ethernet Multicast hashing, but the subject has always beaten me, I never really understood why frame rejection by an imperfect filter could be of much use, until I considered how slow computers were twenty years ago. Neither could I fathom the method used to calculate the correct hash bit from a MAC address. Setting the MAR registers to 0xffffffff causes a network card to receive all multicasts and a software filter achieves the same result, but this week I decided to crack this nut once and for all.
The Multicast bit in an ethernet frame destination address is the first bit placed on the wire by a network transmitter, therefore is the first bit decoded the receivers. So a mechanism exists for rejecting an incoming frame after just ONE BIT. Correct use of the Multicast hash allows a network card to occupy multiple incoming MAC addresses. Though random multicast addresses may hash to the same bit and cause a network interrupt, this can massively reduce the number of receiver interrupts generated for cards attached to a busy wire, and the random frames are easily dropped by software in the next network layer.
Solving the hash bit algorithm was a challenge, as it requires understanding
of ethernet crc32 checksums and the way they relate to the local hardware,
particularly whether the checksum is accumulated by working the data bits
left-to-right or right-to-left. The Multicast hash index may end up in the top
or the bottom 6 bits of the crc32. For the RTL network cards 8029/8136/8100
working on Intel architecture which I am currently programming the hash index
appears in reverse bit order. Placing the hash bit correctly in one of two
32-bit integers (in preparation for a 32-bit port write) is elegantly performed
by:
mar[hash>>5] |= 1 << (hash&0x1f);
Worthy of note is that The MAR registers on the RTL8100 PCIE card are in reverse order
MAR7-MAR0, on the RTL8139 they appear as MAR0-MAR7, just one more trap for the
unwary. The bytes are easily reversed by casting to the ChaOS compiler type
UM (unsigned 32-bit Motorola integer):
outpd(port+0x08,mar[0]); //for RTL8139
outpd(port+0x0c,mar[1]);
outpd(port+0x08,*(UM*)&mar[1]); //for RTL8100
outpd(port+0x0c,*(UM*)&mar[0]);
Testing to prove the above was fun, using the ChaOS debugger to edit the live broadcast MAC address of one computer running CTPPNET, whilst running the same MAC address through the hashing algorithm on a second computer running CTPPNET with broadcasts disabled, multicasts enabled. Seeing the frame arrive with only one bit set in MAR0-7 is impressive. Also worthy of note is that Multicast addresses have bit 0 of byte 0 set to 1. The MAC address burned into a network card always has this bit set to zero.
* diary upload direct from CFS development drive
25/7/2010 HP G62-105 Improved CTPPNET remote drive access, making logical drive range checking conditional on a global variable. This allows CTPPNET to write remote disk sectors anywhere on a mass storage device provided you know how to reset the write lock. Used this tool to backup and replace the Win7 RecoveryMgr partition table with my own, which causes ChaOS to create a small logical drive on hot reboot. Knowing the disk sectors are blank, not in use by Win7 NTFS, I format and sys the logical drive. ChaOS now boots from the notebook hard drive.
I use the EDD ChaOS partition table for this purpose, because it can load a ChaOS image located anywhere on a 2 terabyte drive.
Win7 boot is reinstated by writing the Win7 RecoveryMgr partition table back to LBA 0. There appears to be a backup copy at LBA 2 anyway, so Win7 may use a similar mechanism when switching between Win7, RecoveryMgr and SystemRestore.
Eventually the NTFS partition will overwrite this ChaOS logical drive area, if a system tool such as Norton doesn't delete it, but until that point here is a method for running ChaOS alongside Win7 which is handy enough. I will start some Norton tools on the drive to make sure I am not using a location needed by other ghost software.
Resizing the Win7 NTFS drive should not be too difficult since the Win7 partition is new, and has no Norton installed as yet. This would allow ChaOS a permanent home on a single-drive machine, provided the Win7 installation is new. Internet forums are littered with the wreckage of Windows/Linux dual boot attempts on old NTFS partitions. Nevertheless having ChaOS alongside Win7 is only way to develop a reliable dual-boot ChaOS demo.
* diary upload direct from CFS development drive
23/7/2010 HP G62-105 Booted ChaOS using boot CD on a new HP i3-330m notebook G26-105, to look at a Win7 installation before running it for the first time. Booted straight away though CD drive running BIOS calls was so slow, I thought the beast had hung up. DVD boot then subsequent custom CDs were faster, so the first was probably a poor CD burn. Connection to the LAN and WAN was immediate through RTL8101.DRV (First time Ihave RTL chip id 8103E).
* diary upload direct from CFS development drive
22/7/2010 strcpy() bug! Slight error in NETPATH seen when passing filename with no file extension. The cause is a glaring error when an empty source string is passed to strcpy(). I am generally careful about string handling, often using strlen() followed by memcpy() to be sure strcpy() cannot trash the stack, indeed strcpy() is only called 136 times as ChaOS boots, memcpy() is called 15000 times! Still, no bad thing that I distrust functions which can loop out of control when passed a garbage pointer. Fixed.
22/7/2010 DEC2009.ISO Download Revised DEC2009.ISO (v1.02.26977) uploaded with improved BIOS drive support and strcpy() fix.
* diary upload direct from CFS development drive
21/7/2010 COPYTREE to FAT16 Checking COPYTREE to move current live development (which is on a CFS drive) back to FAT16. All 5500 files are correctly transferred, directory tree is sound, but binary ordering of files may place the root directory entries for the ChaOS bootstrap files outside the first root directory sector. sys() FAT16 currently uses standard file save routines to create the FAT16 bootstrap, which cannot dictate the position of a directory entry so the drive may not boot.
Added setrootentry() function, to search for a given FAT16 root directory 8:3 filename, and if not found in the correct position, swap entries and rewrite to disk. To be sure ChaOS FAT16 bootstrap will always boot after a disk cleanup, added setrootentry("LOADER BIN",0); and setrootentry("CHAOS ",1); to sys() command. Also, to ensure bootstrap can find the device drivers and the banner file, added setrootentry("D ",2); (moves the driver subdirectory) and setrootentry("BANNER ",3);.
* diary upload direct from CFS development drive
19/7/2010 CTPPNET IPX over UDP PATH structure used by generic filesystem commands is just a little too large to place on an unfragmented internet UDP packet along with 512 bytes of filesystem-specific information, so added NETPATH, a compacted version for the network traffic. Added netfindfirst() and netfindnext() to create and despatch the compacted packets, and findfirst(NETPATH*), findnext(NETPATH*) as wrapper functions called by the network redirector to unpack the requests, call the local findfirst(), findnext() functions, then pack the response into NETPATH for the reply packet.
There is space in NETPATH for filesystem-specific information, particularly the findfirst/findnext context which is carried over the network without the redirector needing to understand the internals.
* diary upload direct from CFS development drive
16/7/2010 CTPPNET IPX over UDP Addition of the binary index to CFS means remote CFS drives currently do not work. The FAT16 drives still work, but the mechanism is fairly low-level, and requires a different network command set for each filesystem. Rather than patch up the IPX system, I will add generic commands such as netfindfirst() netfindnext() to get CFS remote drives working again.
A while ago I added IP addresses to the IPX NETUSER structures, in readiness for redirecting the IPX packets via UDP, which could travel through the internet and allow remote handling of ChaOS machines. Added the IPX->UDP redirector today, which works OK, once the UDP packet lengths and checksums are correct.
UDP exchanges over the LAN were more forgiving, with the Vigor router allowing packets with incorrect UDP checksums through to the destination IP. WAN is more demanding. First of all, my router reflects UDP packets with equal source and destination ports back to the sender. This may be because I am using a port in the dynamic range 49152-65535, or it may be a bug. A similar problem exists with the ChaOS PING which I use to reflect a WAN IP address back to the sender. This does not happen when using equal source and destination ports 53 for DNS Query. Choosing dissimilar source and destination ports is the easy workaround. Secondly, the relevant ports need to be opened on the router for the LAN IP to receive the UDP datagram.
The CTPPNET protocol is currently part of the ChaOS kernel, which is not the ideal place. (Is it is loaded before the transport protocols on which the redirector depends!) I get round this by using a dummy ipout() function in the kernel which is hooked by ARP.DRV when it loads. The hook() function locates all relocations to ipout() in the kernel, and maps them to ipout() in ARP. Currently CTPPNET uses a LAN broadcast to advertise a node. WAN broadcasts are not possible in the same way, but NETUSER tables with the correct WAN IP addresses will allow ChaOS machines to connect this way over the internet. The NETUSER table contains both the LAN IP and WAN IP of a node. It will be a simple job to arrange one ChaOS node as the WAN receiver (the node with the open ports on the router). This node can retransmit packets to the correct nodes within the LAN. A little custom encryption will keep the WAN packets safe.
* diary upload direct from CFS development drive
13/7/2010 TCP Addition of out-of-order receiver handling seems to have cured the unreliability in ChaOS FTP STOR across the internet (not surprising - it is part of the original TCP specification!). In essence all the main components of a modern TCP are now present, send-window control, three duplicate acks to cause fast-retransmit, slow-start and congestion window. I added delayed-ack today. What I have not done is slavishly implement all the algorithms in the RFCs, Jacobsen congestion control, round-trip timers and the like, I use trial and error until the code produces the desired result, so the ChaOS TCP is simple staple rather than cordon bleu. TCP enhancements are now held in separate source files, should it be necessary to go back to a recent working version. TCP is 1300 lines of code, compiled as part of IP.DRV.
* diary upload direct from CFS development drive
11/7/2010 TCP Added out-of-order receiver algorithm. Note sure how I got away without it for so long!
11/7/2010 ED As I have reported before, the HTML editor in cPanel is infuriating, being insistent on modifying HTML source code when asked to save a file, yet unable to decode and correct open or cross-nested tags. Gradually the files on this site have been reduced to 80-column format in the ChaOS source editor, indeed this is now the method of choice for diary updates.
Added a simplistic HTML tag highlight to the ChaOS source editor today. Nothing fancy, just scans current display line for < ... > pairs, and colours intervening characters in grey. Makes HTML source much easier visualise, and maintain. <= and -> are special cases, since they occur in C source code.
* diary upload direct from CFS development drive
10/7/2010 FTP ChaOS FTP script mode now working with ChaOS FTPSRV, as well as PureFTPd on Zen server for uploads to websites. For instance upload.ftp might read:
chaos.ctpp.co.uk
user username
pass password
cwd /public_html/chaos
type i
pasv
stor diary.html
quit
ChaOS FTP looks for .ftp file extensions, so FTP upload places the latest version of the diary online in a few seconds.
STOR works fine with Mozilla and WIn Explorer FTP, but STOR to ChaOS FTPSRV is not 100% reliable.
* diary upload direct from CFS development drive
9/7/2010 CFS ChaOS File System/COPYTREE/DISKCOPY Improving CFS partition utilities including copytree and diskcopy, fixed a slight glitch in virtual directory navigation which caused some subdirectory branches to be skipped. By inference, virtual directories do not exist, they are deduced from the boundaries of file sets in the binary index. findfirst() and findnext() will return a virtual directory entry to the caller where file sets exist at deeper branch nesting than the current directory search, to provide compatibility with the behaviour of mainstream filesystems (which would expect to see physical directory entries). These virtual directory entries allow the caller (e.g. FTPSRV for FTP clients) to navigate the filesystem.
DISKCOPY for CFS partitions handles any partition size, provided there is enough space for all the files in the target partition. It does this by adjusting the partition free space record, (there are no file allocation tables or inodes to worry about). The node cache is refreshed and binary index rebuilt, which takes a few seconds before the target partition is ready to use, no reboot required. The layout of the CFS bootstrap is all partition-relative, so if the source partition is CFS-bootable, the target is bootable too.
COPYTREE takes a little longer, but has the advantage that it incorporates file extension set translation. COPYTREE from a CFS partition to an empty CFS partition results in a more compact unit, with files in binary key order on the disk media, better then DEFRAG and faster by two or three orders of magnitude.
9/7/2010 TELSRV Added TIME command to TELSRV, to show current server time and uptime.* diary upload direct from CFS development drive
8/7/2010 CFS ChaOS File System
Decided to try to enable .dot filenames, in order to manage .htaccess via
upload from ChaOS.
In CFS splitpath():
if(*str){ptr=strchr(str,'.');....etc} becomes
if(*str){ptr=strchr(str+1,'.');....etc} and filenames
beginning with a dot are up and running. That was too easy.
* diary upload direct from CFS development drive
7/7/2010 ChaOS UDP/FTPSRV/HTPSRV Added Reboot command yesterday to ChaOS UDP, to allow remote reboot of server at chaos.ctpp.co.uk. Used this for the first time today, after transferring an updated version of HTPSRV.DRV via ChaOS FTP to FTPSRV on the server. Because the ChaOS system reboot is so quick, the new server programs are up and running 4 seconds after the Reboot command is sent.
* diary upload direct from CFS development drive
4/7/2010 ChaOS Diary HTML source for all diary pages painstakingly cleaned, and rendered into 80-column format (instead of the ridiculous 1024-character line-lengths generated by cPanel HTML editor!). Diary pages can now be managed using ChaOS source editor program ED. With ChaOS HTPSRV running on the development machine, changes can be instantly viewed by clicking refresh on a browser running on the LAN. Now HTML files can be tested before being uploaded to the websites. Much easier than writing code to preview my HTML creations. I now have a cheap but fully functional raw HTML editor, with fully-active, full-screen preview, and no PHP code lines were hurt in the making!
* diary upload direct from CFS development drive
3/7/2010 ChaOS IP protocol suite ChaOS DNS rewritten, much simplified and now capable of locating mail server IP from MX record for a given host name.
* diary upload direct from CFS development drive