From elliotnunn at fastmail.com Mon Sep 17 08:44:29 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 17 Sep 2018 08:44:29 +0800 Subject: [CDG5] Hello list! Message-ID: <9028B856-DF6B-4248-BBD1-51B7AB715167@fastmail.com> This email inaugurates the CDG5 mailing list. From elliotnunn at fastmail.com Wed Sep 19 16:33:25 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 19 Sep 2018 16:33:25 +0800 Subject: [CDG5] Newish CubeE build system Message-ID: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> Hi all, I have taken a little break from the NanoKernel and all of those tedious external interrupt handlers. Of course I will come back to it, but currently I am reconsidering the CubeE source tree, and how it can be made to build more reliably. The mac-rom repo currently mashes all sorts of source, binary and build-system changes together, along with plain-English instructions and scripts for making that mess work with empw. Happily, we can currently build a byte-perfect copy of the final NewWorld ROM release. Still, I see problems with this setup. First, it depends on several software components that might go away at any time: - empw itself - minivmac - macOS's non-plus HFS support (currently read-only) - a resource-fork-savvy rsync version - a resource-fork-savvy macOS version - Xcode's MPW-era tools: SetFile, Rez... Second, there are several changes to the makefile structure that help accomodate the binary blobs we use, but that make it difficult to try to build any other ROM versions. Rules have been commented out, empty files created, ad nauseum. (Yes, it was Daniel's email about "Lastly.Lib" actually being "SizeMem.a.o" that sent me off on this tangent.) I propose to produce a piece of software that can run on any modern PC, and that uses only three core technologies to build CubeE: - Python - C (hell, maybe I'll give C++ a try) - a Mac running MPW that can read HFS disk images The general idea is that a single script converts a Git-friendly source tree to an MPW-friendly disk image, allows you to feed that into your emulated system of choice, and then extracts the build results. To get acceptable performance with HFS disk images (unlike empw), direct library calls into hfsutils (https://www.mars.org/home/rob/proj/hfs/) will be necessary. These two repos contain what I've got so far: - https://github.com/elliotnunn/CubeE - https://github.com/elliotnunn/BuildCubeE It would be rather sad if we lost access -- again! -- to the precious ability to build the CubeE sources. Thoughts? Elliot From danielbj314 at verizon.net Wed Sep 19 22:39:42 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Wed, 19 Sep 2018 10:39:42 -0400 Subject: [CDG5] Newish CubeE build system In-Reply-To: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> References: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> Message-ID: <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> Hi Elliot, Sounds pretty good, though I still have a few questions about it. Which ROM versions do you want to build? Definitely NewWorld, maybe OldWorld, possibly all 68k Universal ROMs. Are you going to make a comprehensive change history? What you are doing with the NanoKernel is pretty cool. You might be able to do it here. When you get the time, could you explain the mechanism used to build these big complicated ROMs? I think it goes Build Script -> Something.make (depends on build target) -> all source files -> vectorize -> romlink. Maybe. Thanks, Daniel From elliotnunn at fastmail.com Thu Sep 20 00:46:08 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Thu, 20 Sep 2018 00:46:08 +0800 Subject: [CDG5] Newish CubeE build system In-Reply-To: <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> References: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> Message-ID: Just stuffed hfsutils into a CPython module. Hard slog! I?m hoping to dive further into the NewWorld ROM and get more of it building from source, particularly the parts that seem to wang the interrupt controller. The C compiler used in the build definitely supports inline assembly, so this could be a solution for partly reversing functions that we can?t round-trip. Other than that I?d like to revisit the BlueBox ROM, and see if I can get a near-vanilla CubeE to build (the build tree as is will not build). Slightly different binary versions are very helpful in making sure that you?ve got your reversal right, so a very late NewWorld image might be a good idea. NewWorld was crackable in the first place largely because so much code had been moved out of the ROM. A comprehensive change history is just too much work unless we can automate part of it (and wouldn?t that be cool). Your overview of the build process is pretty close. I?ll describe it in detail a bit later. > On 19 Sep 2018, at 10:39 pm, Daniel B-J wrote: > > Hi Elliot, > > Sounds pretty good, though I still have a few questions about it. > > Which ROM versions do you want to build? Definitely NewWorld, maybe OldWorld, possibly all 68k Universal ROMs. > > Are you going to make a comprehensive change history? What you are doing with the NanoKernel is pretty cool. You might be able to do it here. > > When you get the time, could you explain the mechanism used to build these big complicated ROMs? I think it goes Build Script -> Something.make (depends on build target) -> all source files -> vectorize -> romlink. Maybe. > > Thanks, > Daniel > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Thu Sep 20 00:49:39 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Wed, 19 Sep 2018 12:49:39 -0400 Subject: [CDG5] Newish CubeE build system In-Reply-To: References: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> Message-ID: <5EB3A02D-49F8-4402-B3E7-DE67D6B571A0@verizon.net> Cool. Quick question: Isn?t it like midnight for you right now? From elliotnunn at fastmail.com Thu Sep 20 00:55:17 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Thu, 20 Sep 2018 00:55:17 +0800 Subject: [CDG5] Newish CubeE build system In-Reply-To: <5EB3A02D-49F8-4402-B3E7-DE67D6B571A0@verizon.net> References: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> <5EB3A02D-49F8-4402-B3E7-DE67D6B571A0@verizon.net> Message-ID: <1FA37634-36C2-4F1C-83BB-D8174A74B8F9@fastmail.com> 1am now. I get this week off and I have spent the day hacking. > On 20 Sep 2018, at 12:49 am, daniel B-J wrote: > > Cool. > > Quick question: Isn?t it like midnight for you right now? > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Thu Sep 20 10:50:33 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Thu, 20 Sep 2018 10:50:33 +0800 Subject: [CDG5] CubeE build process In-Reply-To: <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> References: <57558A3E-3427-4905-A7EC-AFEC43DDB1AC@fastmail.com> <103068A6-00CA-4790-BE8C-446ACF051F28@verizon.net> Message-ID: <5BBAF3F6-89EA-4DA1-A483-715A251B727B@fastmail.com> > When you get the time, could you explain the mechanism used to build > these big complicated ROMs? I think it goes Build Script -> > Something.make (depends on build target) -> all source files -> > vectorize -> romlink. Maybe. Step by step... The `:Make:Build` script is a bit crufty. Its job is essentially to run MPW Make, gather its shell-script output, and then run that. It can also override some defines but I steer clear of this. My EasyBuild script is way simpler. Here is an annotated illustration of the dep tree of the RISC makefile. Any of the other three main makefiles will give similar results. StandardEqu.d, a precompiled header used by most of the Asm source files, is not shown. I have had to reimplement Vectorize, RomLayout and RomLink based on various snippets of information. While currently the makefiles are modified to build these from C sources that I put in the repo, the plan is to include them as binaries (like several other things in the Tools folder) and keep the sources elsewhere. This repo was supposed to produce all sorts of bit-sliced and encrypted files for burning to a ROM or running in an emulator. I have only bothered to get the flat-file output working (RomMondo). This file may contain PowerPC code in resources, but it does not contain the NanoKernel or Emulator or any of the other low-level RISC software. RISC builds are only 3 MB. RomMondo RISC.make: this rule only has two deps FeatureSet RISC.make: issues some Set commands :BuildResults:RISC:Image:RomMondo RISC.make: issues a RomLayout command :BuildResults:RISC:Rsrc:RomLayout.Rsrc RISC.make: this is passed as arg to RomLayout :Make:RiscLayout.r RISC.make: Rez file that describes ROM size and paths to resource files below :Resources:RomResources.r Rez file #include'd by RiscLayout.r that lists ROM resources (but not MainCode/DeclData) :Internal:Rez:RomTypes.r :BuildResults:RISC:Rsrc:MainCode.Rsrc MainCode.make: Link creates 'ZROM' rsrc in this file, which is referenced by RomLayout.Rsrc :BuildResults:RISC:Lib:MainCode.Lib MainCode.make: Vectorize'd from several libraries: :BuildResults:RISC:Obj:VectorTablePatch.a.o A special argument to Vectorize that contains the small blobs of glue :Make:VectorTable.a :Internal:Asm:VectorTablePatch.a :BuildResults:RISC:Lib:StartMgr.lib A very generic library within MainCode (the first one) :BuildResults:RISC:Obj:StartTop.a.o :OS:StartMgr:StartTop.a :BuildResults:RISC:Obj:StartInit.a.o :Internal:Asm:HardwarePrivateEqu.a ... :BuildResults:RISC:Obj:VectorTableInit.a.o A rather important library that inits the table that the vector glue uses :Make:VectorTable.a :Internal:Asm:VectorTableInit.a :BuildResults:RISC:Lib:IOPrimitives.lib :BuildResults:RISC:Lib:MMU.lib ... :BuildResults:RISC:Obj:DispTable.a.o The last library in MainCode: inits the A-trap dispatch table with ROM addresses :OS:DispTable.a :Internal:Asm:HardwarePrivateEqu.a :Interfaces:AIncludes:HardwareEqu.a :BuildResults:RISC:Rsrc:EDisk.rsrc ROM resources: this file an example of one built from source... :Misc:GoNativeResources ...and this one is just plonked in the Misc folder... ... ...and there are many more of both types :BuildResults:RISC:Rsrc:DeclData DeclData.make: RomLink decodes the bytecode created by Rez from the source files below :BuildResults:RISC:Rsrc:DeclData.rsrc :Interfaces:RIncludes:Types.r :Internal:Rez:DepVideoEqu.r :Internal:Rez:HardwarePrivateEqu.r :Internal:Rez:InternalOnlyEqu.r :Internal:Rez:QuickDraw.r :Internal:Rez:ROMLink.r :DeclData:DeclData.r From maximumspatium at googlemail.com Mon Sep 24 00:03:35 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Sun, 23 Sep 2018 18:03:35 +0200 Subject: [CDG5] What does PMDT stand for Message-ID: Hello, the abbreviation "PMDT" has already been clarified by Ren? Vega as follows: elliotnunn @ReneVe6a > Still can't find what PMDT stands for. Page Mapping DescripTor? > Replying to @elliotnunn > Page Map Descriptor Table - contains a set of physical address and priv > bits for a range of virtual addresses. Let's fix this comment accordingly: https://github.com/elliotnunn/NanoKernel/blob/3e23e6348f77e5a28ebc7f3589b7c921c008b9d2/Defines.s#L145 Cheers Max -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20180923/c686951c/attachment.html From elliotnunn at fastmail.com Mon Sep 24 06:23:58 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 24 Sep 2018 06:23:58 +0800 Subject: [CDG5] What does PMDT stand for In-Reply-To: References: Message-ID: I have flipped back and forth on this one. Three things make me think that PMDTs might actually be the entries, not the table: - NKv2 prints (plural) ?Converting PMDTs to Areas?, and the correspondence between these two concepts is nearly 1:1 - The flag constants in the early Trampoline are all called PMDT_* - The table already has a consistent name in Apple?s headers: the PageMap > On 24 Sep 2018, at 12:03 am, Max Poliakovski wrote: > > Hello, > > the abbreviation "PMDT" has already been clarified by Ren? Vega as follows: > >> elliotnunn @ReneVe6a >> Still can't find what PMDT stands for. Page Mapping DescripTor? >> >> Replying to @elliotnunn >> Page Map Descriptor Table - contains a set of physical address and priv bits for a range of virtual addresses. > > Let's fix this comment accordingly: https://github.com/elliotnunn/NanoKernel/blob/3e23e6348f77e5a28ebc7f3589b7c921c008b9d2/Defines.s#L145 > > Cheers > Max > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20180924/0dc2487c/attachment.htm