From danielbj314 at verizon.net Sat Dec 1 11:59:33 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Fri, 30 Nov 2018 22:59:33 -0500 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> Message-ID: My "secret project" can do a half-decent job disassembling PEFs. Here's the partial disasm of the 'ncmp' 0 resource, which is called DonnDecompress0. You may or may not get the file, as I have no idea if this mailing list even lets you use attachments. Time to find out. -------------- next part -------------- A non-text attachment was scrubbed... Name: DonnDecompress0.s.sit Type: application/x-stuffit Size: 3321 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181130/b188ca6f/attachment.sit From maximumspatium at googlemail.com Sat Dec 1 22:22:40 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Sat, 1 Dec 2018 15:22:40 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> Message-ID: Hi Daniel, email attachments work. Thank you for sending me your disassembly. From what I see there, that PPC code implements the same compression algorithm by Donn Denman from 1990 as in DeCompressDefProc.a. This becomes apparent when looking at the table of 180 words at the beginning of the dump. The overall algorithm looks like a mix of various well-known lossless techniques: run-length coding (RLE), differential coding and dictionary replacements (both static and dynamic). Code resources will be compressed by replacing the most frequent words found in 68k code with byte indexes. 0x0000 comes first, 0x4EBA (JSR) comes next, followed by 0x0008 and 0x4E75 (RTS) and so on. The whole stuff looks quite easy to understand and to reimplement in Python. I'm prepared to give a try if someone could send me such a compressed resource and its decompressed equivalent. Cheers Max Am Sa., 1. Dez. 2018 um 05:00 Uhr schrieb Daniel B-J < danielbj314 at verizon.net>: > My "secret project" can do a half-decent job disassembling PEFs. > > Here's the partial disasm of the 'ncmp' 0 resource, which is called > DonnDecompress0. > > You may or may not get the file, as I have no idea if this mailing list > even lets you use attachments. Time to find out. > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181201/77afaeb7/attachment.htm From danielbj314 at verizon.net Sat Dec 1 22:58:56 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sat, 1 Dec 2018 09:58:56 -0500 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> Message-ID: <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> > The whole stuff looks quite easy to understand and to reimplement in Python. I'm prepared to give a try if someone could send me such a compressed resource and its decompressed equivalent. Here you go. This may or may not use the compression algorithm I sent you. I don't know how to check that manually. -------------- next part -------------- A non-text attachment was scrubbed... Name: gnld.sit Type: application/x-stuffit Size: 21577 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181201/6f425222/attachment.sit -------------- next part -------------- From elliotnunn at fastmail.com Sun Dec 2 18:27:41 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 2 Dec 2018 18:27:41 +0800 Subject: [CDG5] Random questions In-Reply-To: <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> Message-ID: <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> FWIW, here's another. One of the Process Manager scods from 7.0.1. (Which, as a separate matter, I cheekily binpatched for BuildCubeE) -------------- next part -------------- A non-text attachment was scrubbed... Name: ProcessMgr701_dcmp.zip Type: application/zip Size: 26521 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181202/74f85e6c/attachment.zip -------------- next part -------------- > On 1 Dec 2018, at 10:58 pm, Daniel B-J wrote: > > From elliotnunn at fastmail.com Sun Dec 2 18:43:13 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 2 Dec 2018 18:43:13 +0800 Subject: [CDG5] NanoKernel Message-ID: Help! Hi all, I am back at work on the NanoKernel. The first G3 kernel is close to round-tripping. Then once the early NewWorld kernels are done, the v1 series will be complete. Throughout this version-history project I have tried to produce well-commented sources of high quality. And I have not hesitated to engineer the commit history to make the reversed sources seem eerily prescient. But there are some parts of the kernel on which I have done a bad job, out of concern for time and a lack of understanding of the underlying hardware. The worst pieces are External Interrupts, the RTAS dispatching, some of the MemRetry block and some of the instruction emulation block. So this email is to ask for a bit of help. Max, could you please take a look at the overall history of ExternalInts.s and try to make some sense of it? And Daniel, would please give me your take on RTAS.s, and Emulate.s if you dare? And needless to say, help from others would be appreciated as well! Best, Elliot From jason.duerstock at gmail.com Sun Dec 2 21:08:19 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sun, 2 Dec 2018 08:08:19 -0500 Subject: [CDG5] Random questions In-Reply-To: <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: Here's another random bit I found. Apparently this is the encoding scheme for AppleEvents over TCP/IP: http://www.stuartcheshire.org/papers/COBSforToN.pdf On Sun, Dec 2, 2018 at 5:28 AM Elliot Nunn wrote: > > FWIW, here's another. One of the Process Manager scods from 7.0.1. > > (Which, as a separate matter, I cheekily binpatched for BuildCubeE) > > > > > On 1 Dec 2018, at 10:58 pm, Daniel B-J wrote: > > > > > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Mon Dec 3 07:25:14 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Sun, 2 Dec 2018 18:25:14 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: Message-ID: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> I?m not sure what to make of the RTAS code. For starters, there is only one confirmed usage of the KCallRTAS routine, and it provides little information. https://github.com/elliotnunn/CubeE/blob/master/Toolbox/ShutDownMgr/ShutDownMgr.a I note that there is a cache flushing Special F-Trap, which makes sense as RTAS is supposed to run with paging disabled. Searching the CubeE repo reveals a few other files with the FE20 (KCallRTAS) trap, but most of them are likely just coincidences. Where else would RTAS be used? Maybe the Expansion Bus Manager? We don?t have full sources of that yet. The argument list seems to have the same format as the RTAS implementation in SLOF. https://github.com/aik/SLOF/blob/master/include/rtas.h This seems to be a 64-bit RTAS and OF, but hopefully the interface is the same. I have been unable to locate an actual API specification for RTAS. Any ideas on where to look? Confusingly, the NK implementation doesn?t quite match up with the SLOF version of RTAS. It converts the argument list pointer to a physical address, which makes sense, but then it goes and conditionally does an index with a half-word at offset 0x04. Then some more virt->phys. The same code is in NKv2.28, so it must work. Or are there any systems which actually use RTAS? Apparently the Trampoline can substitute the RTAS for a version contained within it. What does that code look like? > On Dec 2, 2018, at 5:43 AM, Elliot Nunn wrote: > > Help! > > Hi all, > > I am back at work on the NanoKernel. The first G3 kernel is close to > round-tripping. Then once the early NewWorld kernels are done, the v1 > series will be complete. > > Throughout this version-history project I have tried to produce > well-commented sources of high quality. And I have not hesitated to > engineer the commit history to make the reversed sources seem eerily > prescient. But there are some parts of the kernel on which I have done a > bad job, out of concern for time and a lack of understanding of the > underlying hardware. The worst pieces are External Interrupts, the RTAS > dispatching, some of the MemRetry block and some of the instruction > emulation block. So this email is to ask for a bit of help. > > Max, could you please take a look at the overall history of > ExternalInts.s and try to make some sense of it? And Daniel, would > please give me your take on RTAS.s, and Emulate.s if you dare? And > needless to say, help from others would be appreciated as well! > > Best, > > Elliot > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181202/1494b49f/attachment.htm From maximumspatium at googlemail.com Mon Dec 3 08:19:00 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 3 Dec 2018 01:19:00 +0100 Subject: [CDG5] NanoKernel In-Reply-To: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> Message-ID: > I have been unable to locate an actual API specification for RTAS. Any > ideas on where to look? > PowerPC Microprocessor Common Hardware Reference Platform: A System Architecture Chapter 7: Run-Time Abstraction Services https://books.google.de/books?id=H4DNCgAAQBAJ&dq=RTAS+NVRAM_Fetch&hl=de&source=gbs_navlinks_s > The same code is in NKv2.28, so it must work. Or are there any systems > which actually use RTAS? > Apparently the Trampoline can substitute the RTAS for a version contained > within it. What does that code look like? > Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK. The above mentioned libraries will be loaded into the flattened device tree. I don't have any clue when they will be activated. But this happens definitely not in the Trampoline... Cheers Max -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181203/fd0cd11b/attachment.htm From jason.duerstock at gmail.com Mon Dec 3 08:26:26 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sun, 2 Dec 2018 19:26:26 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> Message-ID: Not sure if this is any help or not, but my memory of the PowerPC-related QEMU stuff is that 9.x will boot, but 8.x will not and will crash complaining about missing RTAS bits in the firmware. On Sun, Dec 2, 2018 at 7:20 PM Max Poliakovski wrote: > > >> I have been unable to locate an actual API specification for RTAS. Any ideas on where to look? > > > PowerPC Microprocessor Common Hardware Reference Platform: A System Architecture > Chapter 7: Run-Time Abstraction Services > https://books.google.de/books?id=H4DNCgAAQBAJ&dq=RTAS+NVRAM_Fetch&hl=de&source=gbs_navlinks_s > >> >> The same code is in NKv2.28, so it must work. Or are there any systems which actually use RTAS? >> Apparently the Trampoline can substitute the RTAS for a version contained within it. What does that code look like? > > > Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK. > > The above mentioned libraries will be loaded into the flattened device tree. I don't have any clue when they will be activated. But this happens definitely not in the Trampoline... > > Cheers > Max > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Mon Dec 3 10:50:01 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sun, 2 Dec 2018 21:50:01 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> Message-ID: <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> > Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. It's interesting that they both contain the string "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some undocumented plugin interface? Just for fun, here is dePEF, my MPW tool to disassemble PEFs. Sadly, it's output is not build-able (it does not place csects yet) and it only disassembles PEFs which are at the start of a data fork. It will likely take a few more months of procrastination before it actually is considered adequate by my standards. -------------- next part -------------- A non-text attachment was scrubbed... Name: NV and RTC.sit Type: application/x-stuffit Size: 7204 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181202/2ed4a0ea/attachment.sit -------------- next part -------------- A non-text attachment was scrubbed... Name: dePEF.sit Type: application/x-stuffit Size: 19409 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181202/2ed4a0ea/attachment-0001.sit From jason.duerstock at gmail.com Mon Dec 3 11:20:29 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sun, 2 Dec 2018 22:20:29 -0500 Subject: [CDG5] NanoKernel In-Reply-To: <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: Also, stumbled across this today: https://sourceforge.net/p/mac-on-linux/code/HEAD/tree/trunk/src/drivers/rtas.c On Sun, Dec 2, 2018 at 9:50 PM Daniel B-J wrote: > > > Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK > > Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. It's interesting that they both contain the string "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some undocumented plugin interface? > > Just for fun, here is dePEF, my MPW tool to disassemble PEFs. Sadly, it's output is not build-able (it does not place csects yet) and it only disassembles PEFs which are at the start of a data fork. It will likely take a few more months of procrastination before it actually is considered adequate by my standards. > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Tue Dec 4 02:59:48 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 3 Dec 2018 19:59:48 +0100 Subject: [CDG5] NanoKernel In-Reply-To: <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: Daniel, what disassembler is used using in your dePEF? I noticed that dst and src1 are swapped in some instructions. Below a snippet from the DonnDecompress provided by you: .DonnDecompress: mflr r0 stmw r21,-0x002c(sp) stw r0, 0x0008(sp) stwu sp,-0x0070(sp) stw r3, 0x0088(sp) addi r22,rtoc, 0x0008 *or r5,r26,r5 *---> this should be* or r26, r5, r5 *which corresponds to the simplified form* mr r26, r5* lwz r23,-0x0004(S_0x1_O_0x4) li r0, 0x0001 sth r6, 0x0002(r26) *or r4,r31,r4 --> *should be* or r31, r4, r4* sth r0, 0x0000(r26) Another snippet: S_0x0_O_0x0: lwz r11, 0x0000(r3) lbz r12, 0x0000(r11) addi r10,r11, 0x0001 cmplwi cr0, r12, 0x0080 bt 0,S_0x0_O_0x68 lbz r9, 0x0001(r11) cmpwi cr0, r12, 0x00ff bf 2,S_0x0_O_0x4C lbz r4, 0x0003(r11) *rlwinm r9,r0,24,0,7 --> *should be *rlwinm r0,r9,24,0,7 *to make sense lbz r7, 0x0002(r11) rlwinm r4,r4,8,0,23 lbz r5, 0x0004(r11) rlwinm r7,r7,16,0,15 addi r10,r11, 0x0005 add r6,r5,r4 add r8,r6,r7 add r12,r8,r0 b S_0x0_O_0x68 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181203/2c9b5a08/attachment.htm From maximumspatium at googlemail.com Tue Dec 4 03:03:42 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 3 Dec 2018 20:03:42 +0100 Subject: [CDG5] NanoKernel In-Reply-To: <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: > Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. > It's interesting that they both contain the string > "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some > undocumented plugin interface? > Interesting. I wonder which component depends on these libraries. It's unlikely that NK need them. My assumptions is that they access some hardware from the user mode... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181203/5c76a5cc/attachment.htm From jason.duerstock at gmail.com Tue Dec 4 03:07:16 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Mon, 3 Dec 2018 14:07:16 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: Also, this: http://mirror.informatimago.com/next/developer.apple.com/technotes/tn/tn1192.html On Mon, Dec 3, 2018 at 2:04 PM Max Poliakovski wrote: > > >> Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. It's interesting that they both contain the string "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some undocumented plugin interface? > > > Interesting. I wonder which component depends on these libraries. It's unlikely that NK need them. My assumptions is that they access some hardware from the user mode... > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Tue Dec 4 03:25:30 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 3 Dec 2018 20:25:30 +0100 Subject: [CDG5] Random questions In-Reply-To: <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: Compressed resources have a 18 bytes header at the beginning of the resource data. That header tells some bits about the underlying compression. It has the following format: typedef struct ExtendedResource { uint32_t signature; // must be 0xA89F6572 uint16_t hdrLength; // usually set to 0x12 uint8_t hdrVersion; // 8 - Donn, 9 - Gregg uint8_t extAttributes; // bit 0 == 1 indicates compressed data union { struct { uint8_t varTableRatio; uint8_t overRun; uint16_t decompressID; uint16_t cTableID; } DonnSpecific; struct { uint16_t defprocID; // decompressor resource ID uint16_t decompressSlop; uint8_t byteTabelSize; uint8_t compressFlags; } GreggSpecific; } extHdr; }; The DonnSpecific fields are only valid when hdrVersion == 8. Donn decompression scheme seems to use two distinct algorithms: 0 for data, 1 for code. Looking into the resources you sent to me, I found out that none of them has been compressed using the Donn algo. ProcessManager's scod contains defprocID = 0x0002 while glnd uses defprocID = 0x0003. The former seems to indicate the GreggyBits algo (dcmp 2), while the latter looks like that unknown algo added in OS9 (dcmp 3). Next, I'll try to create a Python decompressor for the Greggy-compressed resource... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181203/3586d2d3/attachment.htm From danielbj314 at verizon.net Tue Dec 4 05:41:06 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Mon, 3 Dec 2018 16:41:06 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: Wrote it myself by looking at PowerPC datasheets. The thing is a big mess of code written entirely by me (with a bit of help from StackOverflow and the like). I guess this is what I get for not reading things close enough. It seems logical instructions have a different operand ordering than arithmetic instructions. Rather than sending a new version over right away, I think I will wait and see if any more of those bugs pop up. > On Dec 3, 2018, at 1:59 PM, Max Poliakovski wrote: > > Daniel, > > what disassembler is used using in your dePEF? I noticed that dst and src1 are swapped in some instructions. Below a snippet from the DonnDecompress provided by you: > > .DonnDecompress: > mflr r0 > stmw r21,-0x002c(sp) > stw r0, 0x0008(sp) > stwu sp,-0x0070(sp) > stw r3, 0x0088(sp) > addi r22,rtoc, 0x0008 > or r5,r26,r5 ---> this should be or r26, r5, r5 which corresponds to the simplified form mr r26, r5 > lwz r23,-0x0004(S_0x1_O_0x4) > li r0, 0x0001 > sth r6, 0x0002(r26) > or r4,r31,r4 --> should be or r31, r4, r4 > sth r0, 0x0000(r26) > > Another snippet: > > S_0x0_O_0x0: > lwz r11, 0x0000(r3) > lbz r12, 0x0000(r11) > addi r10,r11, 0x0001 > cmplwi cr0, r12, 0x0080 > bt 0,S_0x0_O_0x68 > lbz r9, 0x0001(r11) > cmpwi cr0, r12, 0x00ff > bf 2,S_0x0_O_0x4C > lbz r4, 0x0003(r11) > rlwinm r9,r0,24,0,7 --> should be rlwinm r0,r9,24,0,7 to make sense > lbz r7, 0x0002(r11) > rlwinm r4,r4,8,0,23 > lbz r5, 0x0004(r11) > rlwinm r7,r7,16,0,15 > addi r10,r11, 0x0005 > add r6,r5,r4 > add r8,r6,r7 > add r12,r8,r0 > b S_0x0_O_0x68 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181203/9054966d/attachment.htm From maximumspatium at googlemail.com Tue Dec 4 08:08:21 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Tue, 4 Dec 2018 01:08:21 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: The GreggyDecompressor (dcmp 2, https://github.com/elliotnunn/CubeE/blob/master/Patches/GreggyBitsDefProc.a) has been reimplemented in Python (see attachment). Well, this is a partial implementation that is capable of decompressing bitmapped data with embedded lookup table. It produces the same data as found in "Expanded" file for the compressed 'scod' resource I got from Elliot. The only difference is the last four extra bytes I don't know where they come from. The full decompressor need the static lookup table and another simply decoding loop. That's all... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181204/d9f59b85/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: RezDecompress.py Type: text/x-python-script Size: 3853 bytes Desc: not available Url : https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181204/d9f59b85/attachment.bin From maximumspatium at googlemail.com Tue Dec 4 21:59:50 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Tue, 4 Dec 2018 14:59:50 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: I just push my cool GreggDecompressor to Github for you to pick: https://github.com/maximumspatium/ResDecompress It contains all missing parts (predefined lookup table and bytewise decompression) now. Enjoy! > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181204/432d303d/attachment.htm From elliotnunn at fastmail.com Sun Dec 9 11:13:26 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 9 Dec 2018 11:13:26 +0800 Subject: [CDG5] NanoKernel In-Reply-To: <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: I really like this! All the small PEFs in 9.x are the bane of my existence. A simple round-trippable disassembler (which really only needs to support the instructions that MrC used) would be a huge help. Of course the sources would be great. GitHub? And -- any reason dePEF can't be compiled for a modern machine? > On 3 Dec 2018, at 10:50 am, Daniel B-J wrote: > >> Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK > > Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. It's interesting that they both contain the string "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some undocumented plugin interface? > > Just for fun, here is dePEF, my MPW tool to disassemble PEFs. Sadly, it's output is not build-able (it does not place csects yet) and it only disassembles PEFs which are at the start of a data fork. It will likely take a few more months of procrastination before it actually is considered adequate by my standards. > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Sun Dec 9 11:16:51 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 9 Dec 2018 11:16:51 +0800 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: Another question: can, and should, the dePEF change the 'lwz rtoc, 0x14(sp)' that follows cross-fragment function calls back to a 'nop'? > On 9 Dec 2018, at 11:13 am, Elliot Nunn wrote: > > I really like this! All the small PEFs in 9.x are the bane of my > existence. A simple round-trippable disassembler (which really only > needs to support the instructions that MrC used) would be a huge help. > Of course the sources would be great. GitHub? And -- any reason dePEF > can't be compiled for a modern machine? > >> On 3 Dec 2018, at 10:50 am, Daniel B-J wrote: >> >>> Well, the Trampoline will replace OF RTAS with the following three libraries: RTCServicesLib, NVRAMServicesLib and PowerMgrPlugin. These are called "RTAS replacement plug-ins." I think these libraries provide the desired HW abstraction as required by the NK >> >> Here are the partial disassemblies of RTCServicesLib and NVRAMServicesLib. It's interesting that they both contain the string "ThePluginDispatchTable", which is exported by a few parcel PEFs. Some undocumented plugin interface? >> >> Just for fun, here is dePEF, my MPW tool to disassemble PEFs. Sadly, it's output is not build-able (it does not place csects yet) and it only disassembles PEFs which are at the start of a data fork. It will likely take a few more months of procrastination before it actually is considered adequate by my standards. >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Sun Dec 9 12:05:28 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 9 Dec 2018 12:05:28 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: Brilliant! I am considering an extra command line argument for SimpleDeRez: '-decompress'. It would produce .r files like: data 'scod' (123, $40 /*sysheap, _greggybits*/) { ...decompressed data here... }; The '-decompress' option would, like the '-ascii', '-fakehdr' and '-compcmt' options, break the property of producing output identical to the original DeRez. But all of these options would still have the property that when fed into SimpleRez, they would produce a resource file that compressed-resource-aware MacOS finds indistinguishable from the original. And the use of /* _greggybits */ would allow the original Rez to process the file and just output non-compressed resources. This is a lot of work, of course. Re-compression code would be required in addition to Max's de-compression code. Re-compressing identically to the original implementation, like the newworld-rom LZSS code, would be desirable but is not crucial to the round-tripping requirement. And a check for a corresponding 'dcmp' or 'ncmp' resource, assuming that the System file cannot be relied upon, would be appropriate. As soon as recompression is implemented (help Max!!), this feature can be released supporting only GreggyBits for the time being. After all, we will need to allow for obsure third-party 'dcmp's that we have no hope of decompressing. Has anybody identified Mac OS 9's new format? All of this seems like a pain, I know. But imagine how powerful the MakeHFS/DumpHFS workflow would be if we could use it to take any release from (say) the 'Mac OS Anthology', make some patches, and spin up a bootable disk image. > On 4 Dec 2018, at 9:59 pm, Max Poliakovski wrote: > > I just push my cool GreggDecompressor to Github for you to pick: https://github.com/maximumspatium/ResDecompress > > It contains all missing parts (predefined lookup table and bytewise decompression) now. Enjoy! > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Sun Dec 9 22:41:22 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sun, 9 Dec 2018 09:41:22 -0500 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> Message-ID: <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> > A simple round-trippable disassembler (which really only > needs to support the instructions that MrC used) would be a huge help. That's what I'm aiming for. Still haven't reached it yet, but I am getting there. > Of course the sources would be great. GitHub? And -- any reason dePEF > can't be compiled for a modern machine? Here's the sources for dePEF and the 'extensible' framework that it builds on. I cheated on the previous dePEF I sent by merging the 'disasm' file into dePEF. The actual build system doesn't do that for you. Maybe I'll put it on Github once it can be compiled easily on there. Not much point in doing it now. I have no clue how to compile on modern machines. I've done java and python, but that's it. I only really know c on MPW-based systems. > Another question: can, and should, the dePEF change the 'lwz rtoc, > 0x14(sp)' that follows cross-fragment function calls back to a 'nop'? Probably can be made to do so. There are a million things I haven't gotten around to doing with it yet. This is my second time sending this message. For some reason the mailing list didn't notice it the first time. From danielbj314 at verizon.net Sun Dec 9 22:44:47 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sun, 9 Dec 2018 09:44:47 -0500 Subject: [CDG5] NanoKernel In-Reply-To: <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> Message-ID: <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> Also, the mailing list isn't letting me send the actual files for some reason. All the messages that have the files are completely ignored. I have been able to send attachments before, but not this time. Maybe it doesn't like the contents? > On Dec 9, 2018, at 9:41 AM, Daniel B-J wrote: > >> A simple round-trippable disassembler (which really only >> needs to support the instructions that MrC used) would be a huge help. > That's what I'm aiming for. Still haven't reached it yet, but I am getting there. > >> Of course the sources would be great. GitHub? And -- any reason dePEF >> can't be compiled for a modern machine? > Here's the sources for dePEF and the 'extensible' framework that it builds on. I cheated on the previous dePEF I sent by merging the 'disasm' file into dePEF. The actual build system doesn't do that for you. > > Maybe I'll put it on Github once it can be compiled easily on there. Not much point in doing it now. > > I have no clue how to compile on modern machines. I've done java and python, but that's it. I only really know c on MPW-based systems. > >> Another question: can, and should, the dePEF change the 'lwz rtoc, >> 0x14(sp)' that follows cross-fragment function calls back to a 'nop'? > Probably can be made to do so. There are a million things I haven't gotten around to doing with it yet. > > This is my second time sending this message. For some reason the mailing list didn't notice it the first time. > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Mon Dec 10 06:36:58 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 10 Dec 2018 06:36:58 +0800 Subject: [CDG5] NanoKernel In-Reply-To: <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> Message-ID: <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> Too big this time. GitHub! > On 9 Dec 2018, at 10:44 pm, Daniel B-J wrote: > > Also, the mailing list isn't letting me send the actual files for some reason. All the messages that have the files are completely ignored. > I have been able to send attachments before, but not this time. Maybe it doesn't like the contents? >>> On Dec 9, 2018, at 9:41 AM, Daniel B-J wrote: >>> >>> A simple round-trippable disassembler (which really only >>> needs to support the instructions that MrC used) would be a huge help. >> That's what I'm aiming for. Still haven't reached it yet, but I am getting there. >> >>> Of course the sources would be great. GitHub? And -- any reason dePEF >>> can't be compiled for a modern machine? >> Here's the sources for dePEF and the 'extensible' framework that it builds on. I cheated on the previous dePEF I sent by merging the 'disasm' file into dePEF. The actual build system doesn't do that for you. >> >> Maybe I'll put it on Github once it can be compiled easily on there. Not much point in doing it now. >> >> I have no clue how to compile on modern machines. I've done java and python, but that's it. I only really know c on MPW-based systems. >> >>> Another question: can, and should, the dePEF change the 'lwz rtoc, >>> 0x14(sp)' that follows cross-fragment function calls back to a 'nop'? >> Probably can be made to do so. There are a million things I haven't gotten around to doing with it yet. >> >> This is my second time sending this message. For some reason the mailing list didn't notice it the first time. >> >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Mon Dec 10 07:02:53 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sun, 9 Dec 2018 18:02:53 -0500 Subject: [CDG5] NanoKernel In-Reply-To: <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> Message-ID: <562285D7-4583-42CB-91C4-C32405039B98@verizon.net> > Too big this time. GitHub! Done! https://github.com/DBJ314/dePEF-and-disasm -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181209/7d61602c/attachment.htm From elliotnunn at fastmail.com Mon Dec 10 10:43:36 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 10 Dec 2018 10:43:36 +0800 Subject: [CDG5] NanoKernel In-Reply-To: <562285D7-4583-42CB-91C4-C32405039B98@verizon.net> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> <562285D7-4583-42CB-91C4-C32405039B98@verizon.net> Message-ID: <0AC544D2-9504-47D8-81D5-9897F5FF8C6C@fastmail.com> Looks highly sophisticated! Neat! As you know, PEFs have a build time burnt into the header. This is useful in one way: we can make inferences about the Mac OS build system from the embedded build dates (see below). It is also incredibly frustrating, because it prevents us from making reproducible builds. (This seems to be a bit of a topic among security people: https://reproducible-builds.org) So far I have half-solved the problem in BuildCubeE by setting the Ticks lowmem variable at build-box boot, and hoping that the build completes before the emulator notices and corrects the time. Even when it works, it is only good enough for the date-only DeclData stamp. Any thoughts on this problem? I reckon we will have to wrap Rez in a source-preprocessing script, and PPCAsm in a binary-postprocessing script. 2002-07-16 12:30:05 [parcels/pciclass,0c0010:ieee1394 (name:DFMLib)] 2002-07-16 12:30:07 [parcels/pciclass,0c0010:ieee1394 (name:DevNLib)] 2002-07-16 12:30:08 [parcels/pciclass,0c0010:ieee1394 (name:FWServicesLib)] 2002-07-16 12:30:19 [parcels/pciclass,0c0010:ieee1394 (name:GenericDriverFamily)] 2002-07-16 12:30:34 [parcels/pciclass,0c0010:ieee1394 (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 22:26:27 [parcels/CodeRegister Node Parcel (name:EtherPrintfLib)] 2002-09-03 22:31:22 [parcels/CodeRegister Node Parcel (name:StartLib)] 2002-09-03 22:37:18 [/tmp/mac-os-rom-9.6.1.rsrc] 2002-09-03 22:55:14 [parcels/pci104c,ac1a:cardbus (name:PCCard)] 2002-09-03 22:55:14 [parcels/pci104c,ac50:cardbus (name:PCCard)] 2002-09-03 22:56:47 [parcels/cuda:via-cuda (name:pef,AAPL,MacOS,PowerPC,register)] 2002-09-03 22:56:56 [parcels/pmu:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] 2002-09-03 22:57:13 [parcels/via-pmu-99:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] 2002-09-03 22:57:27 [parcels/via-pmu-2000:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] 2002-09-03 22:58:01 [parcels/CodeRegister Node Parcel (name:NativePowerMgrLib)] 2002-09-03 22:58:10 [parcels/cofb:display (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 22:58:33 [parcels/CodePrepare Node Parcel (name:TimeManagerLib)] 2002-09-03 23:00:16 [parcels/Mac OS ROM Parcel] 2002-09-03 23:01:07 [parcels/backlight:backlight (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:01:18 [parcels/cmd646-ata:ata (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:01:22 [parcels/nvram,flash:nvram (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:01:24 [parcels/mac-io:nvram (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:01:28 [parcels/via-cuda:rtc (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:01:29 [parcels/grackle:pci (name:pef,AAPL,MacOS,PowerPC,prepare)] 2002-09-03 23:01:47 [parcels/heathrow-ata:ata,ide (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:02:06 [parcels/keylargo-ata:ata (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:02:25 [parcels/kauai-ata:ata (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:02:34 [parcels/CodePrepare Node Parcel (name:NVRAMServicesLib)] 2002-09-03 23:02:37 [parcels/pmu:power-mgt (name:PMULib)] 2002-09-03 23:02:39 [parcels/via-pmu:nvram (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:02:41 [parcels/via-pmu:rtc (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:02:48 [parcels/CodePrepare Node Parcel (name:RTCServicesLib)] 2002-09-03 23:03:00 [parcels/pci104c,ac1a:cardbus (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:03:02 [parcels/uni-north:pci (name:pef,AAPL,MacOS,PowerPC,prepare)] 2002-09-03 23:03:04 [parcels/CodeRegister Node Parcel (name:AGPLib)] 2002-09-03 23:03:07 [parcels/keywest-i2c:i2c (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:03:07 [parcels/uni-n-i2c:i2c (name:driver,AAPL,MacOS,PowerPC)] 2002-09-03 23:03:13 [parcels/bmac+:network (name:lanLib,AAPL,MacOS,PowerPC)] 2002-09-03 23:03:18 [parcels/apple21143:network (name:lanLib,AAPL,MacOS,PowerPC)] 2002-09-03 23:03:24 [parcels/gmac:network (name:lanLib,AAPL,MacOS,PowerPC)] > On 10 Dec 2018, at 7:02 am, Daniel B-J wrote: > >> Too big this time. GitHub! > > Done! https://github.com/DBJ314/dePEF-and-disasm > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From jason.duerstock at gmail.com Mon Dec 10 10:53:41 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sun, 9 Dec 2018 21:53:41 -0500 Subject: [CDG5] NanoKernel In-Reply-To: <0AC544D2-9504-47D8-81D5-9897F5FF8C6C@fastmail.com> References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> <562285D7-4583-42CB-91C4-C32405039B98@verizon.net> <0AC544D2-9504-47D8-81D5-9897F5FF8C6C@fastmail.com> Message-ID: You could just write a simple "peftouch" program or something. On Sun, Dec 9, 2018 at 9:44 PM Elliot Nunn wrote: > > Looks highly sophisticated! Neat! > > As you know, PEFs have a build time burnt into the header. This is > useful in one way: we can make inferences about the Mac OS build system > from the embedded build dates (see below). It is also incredibly > frustrating, because it prevents us from making reproducible builds. > (This seems to be a bit of a topic among security people: > https://reproducible-builds.org) > > So far I have half-solved the problem in BuildCubeE by setting the Ticks > lowmem variable at build-box boot, and hoping that the build completes > before the emulator notices and corrects the time. Even when it works, > it is only good enough for the date-only DeclData stamp. > > Any thoughts on this problem? I reckon we will have to wrap Rez in a > source-preprocessing script, and PPCAsm in a binary-postprocessing > script. > > 2002-07-16 12:30:05 [parcels/pciclass,0c0010:ieee1394 (name:DFMLib)] > 2002-07-16 12:30:07 [parcels/pciclass,0c0010:ieee1394 (name:DevNLib)] > 2002-07-16 12:30:08 [parcels/pciclass,0c0010:ieee1394 (name:FWServicesLib)] > 2002-07-16 12:30:19 [parcels/pciclass,0c0010:ieee1394 (name:GenericDriverFamily)] > 2002-07-16 12:30:34 [parcels/pciclass,0c0010:ieee1394 (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 22:26:27 [parcels/CodeRegister Node Parcel (name:EtherPrintfLib)] > 2002-09-03 22:31:22 [parcels/CodeRegister Node Parcel (name:StartLib)] > 2002-09-03 22:37:18 [/tmp/mac-os-rom-9.6.1.rsrc] > 2002-09-03 22:55:14 [parcels/pci104c,ac1a:cardbus (name:PCCard)] > 2002-09-03 22:55:14 [parcels/pci104c,ac50:cardbus (name:PCCard)] > 2002-09-03 22:56:47 [parcels/cuda:via-cuda (name:pef,AAPL,MacOS,PowerPC,register)] > 2002-09-03 22:56:56 [parcels/pmu:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] > 2002-09-03 22:57:13 [parcels/via-pmu-99:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] > 2002-09-03 22:57:27 [parcels/via-pmu-2000:power-mgt (name:pef,AAPL,MacOS,PowerPC,register)] > 2002-09-03 22:58:01 [parcels/CodeRegister Node Parcel (name:NativePowerMgrLib)] > 2002-09-03 22:58:10 [parcels/cofb:display (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 22:58:33 [parcels/CodePrepare Node Parcel (name:TimeManagerLib)] > 2002-09-03 23:00:16 [parcels/Mac OS ROM Parcel] > 2002-09-03 23:01:07 [parcels/backlight:backlight (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:01:18 [parcels/cmd646-ata:ata (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:01:22 [parcels/nvram,flash:nvram (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:01:24 [parcels/mac-io:nvram (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:01:28 [parcels/via-cuda:rtc (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:01:29 [parcels/grackle:pci (name:pef,AAPL,MacOS,PowerPC,prepare)] > 2002-09-03 23:01:47 [parcels/heathrow-ata:ata,ide (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:02:06 [parcels/keylargo-ata:ata (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:02:25 [parcels/kauai-ata:ata (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:02:34 [parcels/CodePrepare Node Parcel (name:NVRAMServicesLib)] > 2002-09-03 23:02:37 [parcels/pmu:power-mgt (name:PMULib)] > 2002-09-03 23:02:39 [parcels/via-pmu:nvram (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:02:41 [parcels/via-pmu:rtc (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:02:48 [parcels/CodePrepare Node Parcel (name:RTCServicesLib)] > 2002-09-03 23:03:00 [parcels/pci104c,ac1a:cardbus (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:03:02 [parcels/uni-north:pci (name:pef,AAPL,MacOS,PowerPC,prepare)] > 2002-09-03 23:03:04 [parcels/CodeRegister Node Parcel (name:AGPLib)] > 2002-09-03 23:03:07 [parcels/keywest-i2c:i2c (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:03:07 [parcels/uni-n-i2c:i2c (name:driver,AAPL,MacOS,PowerPC)] > 2002-09-03 23:03:13 [parcels/bmac+:network (name:lanLib,AAPL,MacOS,PowerPC)] > 2002-09-03 23:03:18 [parcels/apple21143:network (name:lanLib,AAPL,MacOS,PowerPC)] > 2002-09-03 23:03:24 [parcels/gmac:network (name:lanLib,AAPL,MacOS,PowerPC)] > > > On 10 Dec 2018, at 7:02 am, Daniel B-J wrote: > > > >> Too big this time. GitHub! > > > > Done! https://github.com/DBJ314/dePEF-and-disasm > > _______________________________________________ > > cdg5 mailing list > > cdg5 at ucc.asn.au > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Mon Dec 10 20:58:33 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 10 Dec 2018 13:58:33 +0100 Subject: [CDG5] NanoKernel In-Reply-To: References: <45EE89AB-22BB-4997-9BEF-5280BE4203F8@verizon.net> <30E9A584-CD55-4C9E-8AC5-E047FB64485C@verizon.net> <168CF541-B677-42AF-A11C-CC0606D863B6@verizon.net> <8A151C9B-9ED5-4E97-9D03-DB41E01E9487@verizon.net> <9F3340AA-26FA-4A5C-9272-B1EAD119457B@fastmail.com> <562285D7-4583-42CB-91C4-C32405039B98@verizon.net> <0AC544D2-9504-47D8-81D5-9897F5FF8C6C@fastmail.com> Message-ID: Stuffing the desired build date into the PEF header during post-processing is also my preferable solution... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181210/111031f1/attachment.htm From maximumspatium at googlemail.com Mon Dec 10 21:24:27 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Mon, 10 Dec 2018 14:24:27 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: MacOS 9 doesn't apparently use compressed resources. I looked into the resource map of the System file from MacOS 9.2.2 and couldn't find any resource starting with 0xA89F6572. Moreover, dcmp 3 does exist as 68k code only while Donn&Gregg decompression methods have PPC equivalents in 'ncmp' resources. I therefore need more compressed resources for testing purpose. It would be great I you could locate Donn-compressed resources as well. They usually start with 0xA89F6572 00XX 0801... Regarding re-compression, I think it should be easy to build a compressor for GreggyBits controlled by the decompression params. I'm not sure if the original compressor (whose source isn't publicly available) had an automatic way of determining best-fit compression params... BTW, the author of GreggyBits compression - Greg Marriott - still does software development. According to his blog http://www.greggybits.com/about.html, he works for Apple again. He'd be surely pleased to see his ol' stuff reimplemented in Python. I'd do in his place... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181210/5aed4d99/attachment.htm From elliotnunn at fastmail.com Mon Dec 10 23:06:47 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 10 Dec 2018 23:06:47 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> If you have the time, a Greggy compressor would be more immediately useful than a Donn decompressor. > On 10 Dec 2018, at 9:24 pm, Max Poliakovski wrote: > > Regarding re-compression, I think it should be easy to build a compressor for GreggyBits controlled by the decompression params. I'm not sure if the original compressor (whose source isn't publicly available) had an automatic way of determining best-fit compression params... > From elliotnunn at fastmail.com Mon Dec 10 23:10:06 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 10 Dec 2018 23:10:06 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> Message-ID: <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> Grepping through System 9.2.2 reveals several compressed resources: $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ -- data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ Could you have used a search method that extracted the resources automatically? Here are some Donn resources, both compressed and extracted (manual operation... ugh!). How will I identify the mysterious Mac OS 9 format? https://github.com/elliotnunn/donn-examples Good for Greg! > On 10 Dec 2018, at 9:24 pm, Max Poliakovski wrote: > > MacOS 9 doesn't apparently use compressed resources. I looked into the resource map of the System file from MacOS 9.2.2 and couldn't find any resource starting with 0xA89F6572. > > Moreover, dcmp 3 does exist as 68k code only while Donn&Gregg decompression methods have PPC equivalents in 'ncmp' resources. > > I therefore need more compressed resources for testing purpose. It would be great I you could locate Donn-compressed resources as well. They usually start with 0xA89F6572 00XX 0801... > > Regarding re-compression, I think it should be easy to build a compressor for GreggyBits controlled by the decompression params. I'm not sure if the original compressor (whose source isn't publicly available) had an automatic way of determining best-fit compression params... > > BTW, the author of GreggyBits compression - Greg Marriott - still does software development. According to his blog http://www.greggybits.com/about.html, he works for Apple again. He'd be surely pleased to see his ol' stuff reimplemented in Python. I'd do in his place... > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Tue Dec 11 17:45:13 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Tue, 11 Dec 2018 17:45:13 +0800 Subject: [CDG5] Random questions In-Reply-To: <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> Message-ID: I have prepared macresources for 'proper' compression support in a new branch. Currently the actual decompression/recompression is faked out by this file. Max, is it possible for you to wrap your code in a Python package (folder with an __init__.py) that exports a similar API, for inclusion in the macresources repo? https://github.com/elliotnunn/macresources/blob/compression/macresources/FakeResDecompress.py Regrettably the System 7 resource compression scheme was a bit messy. Additionally, Rez and DeRez treat compressed resources a bit oddly because those tools do not call the Resource Manager (for 'historical' reasons, likely the Lisa). These two constraints have forced me to implement a quite complicated algorithm for parsing and representing compressed resources. I plan to expose this code path via the '-expand' argument to SimpleDeRez. Currently it does these things: - Exposes decompressed data via the Resource.data attribute. If the data is in an unsupported format the access fails unless the user sets Resource.expand_err_ok. This error is also suppressed when trying to extract a resource for saving to a Rez file, because in that case the Rez code generator will cancel the decompression and leave the 'compressed' flag set. - Lazily decompresses and recompresses a resource only when absolutely necessary. - No matter what, Rez or SimpleRez will always be able to slurp a file I have generated, and produce a resource fork that contains correct data when accessed via the Resource Manager (although it may no longer be compressed). SimpleRez will recognise commented annotations like /* Compress: DonnBits */ and do the needful, while Rez will naturally ignore them. - It is always safe for an API user to change the Resource.compression_format attribute, say to 'DonnBits' or to None. Care is taken that while binary data can be *converted* between formats, it will never be *interpreted* in the wrong format. To clarify the first point a bit, here are some of the Rez-language outputs that might be produced when extraction is attempted: - data 'abcd' (1, $01) /* Unknown compression format */ ... followed by compressed data (note the compressed bit is set in $01) - data 'abcd' (1) /* Compress: DonnBits */ ... followed by expanded data (which SimpleRez will recompress, but but Rez won't) By the way, I think it was Jason who first got us onto the topic of compressed resources. Good thinking man! This is an idea whose time has come, because I have some plans for the System size of the CubeE repo, compressed resources and all. Elliot > On 10 Dec 2018, at 11:10 pm, Elliot Nunn wrote: > > Grepping through System 9.2.2 reveals several compressed resources: > > $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 > data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { > $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ > $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ > $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ > -- > data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { > $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ > $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ > $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ > > Could you have used a search method that extracted the resources > automatically? > > Here are some Donn resources, both compressed and extracted (manual > operation... ugh!). How will I identify the mysterious Mac OS 9 format? > > https://github.com/elliotnunn/donn-examples > > Good for Greg! > >> On 10 Dec 2018, at 9:24 pm, Max Poliakovski wrote: >> >> MacOS 9 doesn't apparently use compressed resources. I looked into the resource map of the System file from MacOS 9.2.2 and couldn't find any resource starting with 0xA89F6572. >> >> Moreover, dcmp 3 does exist as 68k code only while Donn&Gregg decompression methods have PPC equivalents in 'ncmp' resources. >> >> I therefore need more compressed resources for testing purpose. It would be great I you could locate Donn-compressed resources as well. They usually start with 0xA89F6572 00XX 0801... >> >> Regarding re-compression, I think it should be easy to build a compressor for GreggyBits controlled by the decompression params. I'm not sure if the original compressor (whose source isn't publicly available) had an automatic way of determining best-fit compression params... >> >> BTW, the author of GreggyBits compression - Greg Marriott - still does software development. According to his blog http://www.greggybits.com/about.html, he works for Apple again. He'd be surely pleased to see his ol' stuff reimplemented in Python. I'd do in his place... >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > From maximumspatium at googlemail.com Wed Dec 12 00:24:07 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Tue, 11 Dec 2018 17:24:07 +0100 Subject: [CDG5] Random questions In-Reply-To: <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> Message-ID: Hi Elliot, If you have the time, a Greggy compressor would be more immediately > useful than a Donn decompressor. > With a bit of Python magic, my tiny tool now supports resource compression according to the Gregg scheme, see this commit: https://github.com/maximumspatium/ResDecompress/commit/6c81808433d9eb6c696e330081c2c425add4d351 The re-compressed output is bit-identical to the original in the "Compressed" file. Enjoy! Cheers Max -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181211/8673951f/attachment.htm From elliotnunn at fastmail.com Wed Dec 12 07:24:33 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 12 Dec 2018 07:24:33 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> Message-ID: <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> Neat! What is the significance of isBitMapped? > On 12 Dec 2018, at 12:24 am, Max Poliakovski wrote: > > Hi Elliot, > >> If you have the time, a Greggy compressor would be more immediately >> useful than a Donn decompressor. > > With a bit of Python magic, my tiny tool now supports resource compression according to the Gregg scheme, see this commit: https://github.com/maximumspatium/ResDecompress/commit/6c81808433d9eb6c696e330081c2c425add4d351 > > The re-compressed output is bit-identical to the original in the "Compressed" file. Enjoy! > > Cheers > Max > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/b243a0cd/attachment.htm From maximumspatium at googlemail.com Wed Dec 12 10:17:11 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Wed, 12 Dec 2018 03:17:11 +0100 Subject: [CDG5] Random questions In-Reply-To: <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> Message-ID: >From my understanding, the bit-mapped compression allows for a bigger alphabet than a 256 words table can hold. This means a greater flexibility but a worse compression ratio. Instead of replacing each word with a table index (which ideally yields a 2:1 compression), we add an extra byte for each group of eight words indicating if a particular word is coded "as is" (plain text) or as table index. The compression ratio in this case is left as an exercise to the reader. Non-bitmapped compression assumes that all words will be in the lookup table that, in turn, seems to be pretty small for large resources. This therefore makes me think that the bitmapped compression is used more frequently than the non-bitmapped one. My code doesn't currently support automatic mode/table switching but relies on these parameters being supplied by user. Because we're speaking about small changes, recompressing a particular resource under control of the original parameters looks like the easiest solution... Am Mi., 12. Dez. 2018 um 00:25 Uhr schrieb Elliot Nunn < elliotnunn at fastmail.com>: > Neat! > > What is the significance of isBitMapped? > ___________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/39f380d1/attachment.htm From elliotnunn at fastmail.com Wed Dec 12 11:20:59 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 12 Dec 2018 11:20:59 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> Message-ID: Cool! The code I?ve written allows compression algorithms to be described by an arbitrary string. ?GreggyBitsBitmap? would work? > On 12 Dec 2018, at 10:17 am, Max Poliakovski wrote: > > From my understanding, the bit-mapped compression allows for a bigger alphabet than a 256 words table can hold. This means a greater flexibility but a worse compression ratio. Instead of replacing each word with a table index (which ideally yields a 2:1 compression), we add an extra byte for each group of eight words indicating if a particular word is coded "as is" (plain text) or as table index. The compression ratio in this case is left as an exercise to the reader. > > Non-bitmapped compression assumes that all words will be in the lookup table that, in turn, seems to be pretty small for large resources. This therefore makes me think that the bitmapped compression is used more frequently than the non-bitmapped one. > > My code doesn't currently support automatic mode/table switching but relies on these parameters being supplied by user. Because we're speaking about small changes, recompressing a particular resource under control of the original parameters looks like the easiest solution... > > >> Am Mi., 12. Dez. 2018 um 00:25 Uhr schrieb Elliot Nunn : >> Neat! >> >> What is the significance of isBitMapped? >> ___________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/447d75e3/attachment.htm From elliotnunn at fastmail.com Wed Dec 12 11:50:13 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 12 Dec 2018 11:50:13 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> Message-ID: On reflection, perhaps auto detection would be better, and quicker to implement. And it covers the case where a resource gets changed to become inexpressible by the bitmap method. > On 12 Dec 2018, at 11:20 am, Elliot Nunn wrote: > > Cool! The code I?ve written allows compression algorithms to be described by an arbitrary string. ?GreggyBitsBitmap? would work? > >> On 12 Dec 2018, at 10:17 am, Max Poliakovski wrote: >> >> From my understanding, the bit-mapped compression allows for a bigger alphabet than a 256 words table can hold. This means a greater flexibility but a worse compression ratio. Instead of replacing each word with a table index (which ideally yields a 2:1 compression), we add an extra byte for each group of eight words indicating if a particular word is coded "as is" (plain text) or as table index. The compression ratio in this case is left as an exercise to the reader. >> >> Non-bitmapped compression assumes that all words will be in the lookup table that, in turn, seems to be pretty small for large resources. This therefore makes me think that the bitmapped compression is used more frequently than the non-bitmapped one. >> >> My code doesn't currently support automatic mode/table switching but relies on these parameters being supplied by user. Because we're speaking about small changes, recompressing a particular resource under control of the original parameters looks like the easiest solution... >> >> >>> Am Mi., 12. Dez. 2018 um 00:25 Uhr schrieb Elliot Nunn : >>> Neat! >>> >>> What is the significance of isBitMapped? >>> ___________________________________ >>> cdg5 mailing list >>> cdg5 at ucc.asn.au >>> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/52f63930/attachment.htm From maximumspatium at googlemail.com Wed Dec 12 16:47:31 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Wed, 12 Dec 2018 09:47:31 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <6C3017B0-3512-4ABA-B4F2-3ADB3428AF54@fastmail.com> <86DA12C6-E8CA-4BEC-A718-5C788FF75FC8@fastmail.com> Message-ID: The bitmap method is capable of expressing any input. Its output isn't guarantied to be optimal though. In the theoretical worst case, the compressed string will be longer than the uncompressed one. This may happen for highly randomized data, although I don't know that for sure. Auto-selection of the compression method is indeed easy in this precise case. If the derived words dictionary contains > 256 entries, select the bitmap method (i.e. extended alphabet), otherwise - the non-bitmap method. The latter is guarantied to achieve a better compression. As opposite, I don't know how to quickly perform auto-selection of coding tables without multi-pass encoding and comparing lengths. That's the point where some additional tests on real-world data may shed some light... Am Mi., 12. Dez. 2018 um 05:01 Uhr schrieb Elliot Nunn < elliotnunn at fastmail.com>: > On reflection, perhaps auto detection would be better, and quicker to > implement. And it covers the case where a resource gets changed to become > inexpressible by the bitmap method. > > On 12 Dec 2018, at 11:20 am, Elliot Nunn wrote: > > Cool! The code I?ve written allows compression algorithms to be described > by an arbitrary string. ?GreggyBitsBitmap? would work? > > On 12 Dec 2018, at 10:17 am, Max Poliakovski < > maximumspatium at googlemail.com> wrote: > > From my understanding, the bit-mapped compression allows for a bigger > alphabet than a 256 words table can hold. This means a greater flexibility > but a worse compression ratio. Instead of replacing each word with a table > index (which ideally yields a 2:1 compression), we add an extra byte for > each group of eight words indicating if a particular word is coded "as is" > (plain text) or as table index. The compression ratio in this case is left > as an exercise to the reader. > > Non-bitmapped compression assumes that all words will be in the lookup > table that, in turn, seems to be pretty small for large resources. This > therefore makes me think that the bitmapped compression is used more > frequently than the non-bitmapped one. > > My code doesn't currently support automatic mode/table switching but > relies on these parameters being supplied by user. Because we're speaking > about small changes, recompressing a particular resource under control of > the original parameters looks like the easiest solution... > > > Am Mi., 12. Dez. 2018 um 00:25 Uhr schrieb Elliot Nunn < > elliotnunn at fastmail.com>: > >> Neat! >> >> What is the significance of isBitMapped? >> ___________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >> > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/4b35472d/attachment.htm From maximumspatium at googlemail.com Wed Dec 12 17:35:32 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Wed, 12 Dec 2018 10:35:32 +0100 Subject: [CDG5] Random questions In-Reply-To: <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> Message-ID: Elliot, Grepping through System 9.2.2 reveals several compressed resources: > > Could you have used a search method that extracted the resources > automatically? > No. I opened the resource fork in a Unix Hexeditor and searched for 0xA89F6572. According to ResEdit, both CDEF #62 and DRVR #2 are uncompressed in my german "System" file: the "compressed" checkbox isn't selected. Does ResEdit unpack resources in background when opening a file for edit? $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ -- data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ > Here are some Donn resources, both compressed and extracted (manual > operation... ugh!). How will I identify the mysterious Mac OS 9 format? > CDEF #62: $"A89F 6572 0012 0901 0000 0454 *0003* 03FF" DRVR #2: $"A89F 6572 0012 0901 0000 019A *0003* 03FF" According to the defconID = 0x0003, both resources are compressed with that mysterious new method. A short peek into 'dcmp' #3 reveals a sophisticated compression method utilizing Huffman coding. It contains neither strings nor tables. Just a lot of code, much larger than Donn&Gregg stuff. I therefore suggest to postpone its RE until the Trampoline work is finished... BTW, many thanks for the Donn bits! I'll analyze them soon... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/f6ff7540/attachment.htm From elliotnunn at fastmail.com Wed Dec 12 23:03:13 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 12 Dec 2018 23:03:13 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> Message-ID: <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Huh... that?s surprising news about your System file. Given the largely transparent nature of this compression scheme, it is possible to imagine accidental expansion occurring somewhere in the build, localisation, installation or update process. But I don?t know. Good call on the Huffman code. Any chance it could be a well known algo? Is there a 68k implementation? > On 12 Dec 2018, at 5:35 pm, Max Poliakovski wrote: > > Elliot, > >> Grepping through System 9.2.2 reveals several compressed resources: >> >> Could you have used a search method that extracted the resources >> automatically? > > No. I opened the resource fork in a Unix Hexeditor and searched for 0xA89F6572. > According to ResEdit, both CDEF #62 and DRVR #2 are uncompressed in my german "System" file: the "compressed" checkbox isn't selected. > Does ResEdit unpack resources in background when opening a file for edit? > > $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 > data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { > $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ > $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ > $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ > -- > data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { > $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ > $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ > $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ > >> Here are some Donn resources, both compressed and extracted (manual >> operation... ugh!). How will I identify the mysterious Mac OS 9 format? > > CDEF #62: > $"A89F 6572 0012 0901 0000 0454 0003 03FF" > DRVR #2: > $"A89F 6572 0012 0901 0000 019A 0003 03FF" > > According to the defconID = 0x0003, both resources are compressed with that mysterious new method. > > A short peek into 'dcmp' #3 reveals a sophisticated compression method utilizing Huffman coding. It contains neither strings nor tables. Just a lot of code, much larger than Donn&Gregg stuff. I therefore suggest to postpone its RE until the Trampoline work is finished... > > BTW, many thanks for the Donn bits! I'll analyze them soon... > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181212/96ebc402/attachment.htm From jason.duerstock at gmail.com Thu Dec 13 07:44:19 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Wed, 12 Dec 2018 18:44:19 -0500 Subject: [CDG5] Random questions In-Reply-To: <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: There's only a 68k implementation (dcmp.0003), no ppc implementation. On Wed, Dec 12, 2018 at 10:03 AM Elliot Nunn wrote: > > Huh... that?s surprising news about your System file. Given the largely transparent nature of this compression scheme, it is possible to imagine accidental expansion occurring somewhere in the build, localisation, installation or update process. But I don?t know. > > Good call on the Huffman code. Any chance it could be a well known algo? Is there a 68k implementation? > > On 12 Dec 2018, at 5:35 pm, Max Poliakovski wrote: > > Elliot, > >> Grepping through System 9.2.2 reveals several compressed resources: >> >> Could you have used a search method that extracted the resources >> automatically? > > > No. I opened the resource fork in a Unix Hexeditor and searched for 0xA89F6572. > According to ResEdit, both CDEF #62 and DRVR #2 are uncompressed in my german "System" file: the "compressed" checkbox isn't selected. > Does ResEdit unpack resources in background when opening a file for edit? > > $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 > data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { > $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ > $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ > $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ > -- > data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { > $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ > $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ > $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ > >> >> Here are some Donn resources, both compressed and extracted (manual >> operation... ugh!). How will I identify the mysterious Mac OS 9 format? > > > CDEF #62: > $"A89F 6572 0012 0901 0000 0454 0003 03FF" > DRVR #2: > $"A89F 6572 0012 0901 0000 019A 0003 03FF" > > According to the defconID = 0x0003, both resources are compressed with that mysterious new method. > > A short peek into 'dcmp' #3 reveals a sophisticated compression method utilizing Huffman coding. It contains neither strings nor tables. Just a lot of code, much larger than Donn&Gregg stuff. I therefore suggest to postpone its RE until the Trampoline work is finished... > > BTW, many thanks for the Donn bits! I'll analyze them soon... > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Thu Dec 13 17:21:53 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Thu, 13 Dec 2018 17:21:53 +0800 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: <2D83A360-C331-4463-985D-B886EBB9D587@fastmail.com> Oops! I got that one backwards. My 9.2.2 System file has plenty of Huffmanesque resources now that I have a look at it. Max, I'm thinking about how to package macresources, now that my 'compression' branch is mature enough to round-trip everything I throw at it. In particular, it handles the 'missing algorithm' case elegantly, so we can add algorithms at will. There are a few ways to call your code from my PyPI package (you'll see a pull request on GitHub with some API changes): 1. Package your own on PyPI so I can declare it as a dependency. 2. Merge it Git-wise into my repo, and then I merge your commits as I go. 3. Merge it as above, but then do future development in the macresources repo. I favour option 3 as the most natural way. ResDecompress will be exported as a subpackage (maybe with a lowercase name). Happy with the MIT license on macresources? > On 13 Dec 2018, at 7:44 am, Jason Duerstock wrote: > > There's only a 68k implementation (dcmp.0003), no ppc implementation. > On Wed, Dec 12, 2018 at 10:03 AM Elliot Nunn wrote: >> >> Huh... that?s surprising news about your System file. Given the largely transparent nature of this compression scheme, it is possible to imagine accidental expansion occurring somewhere in the build, localisation, installation or update process. But I don?t know. >> >> Good call on the Huffman code. Any chance it could be a well known algo? Is there a 68k implementation? >> >> On 12 Dec 2018, at 5:35 pm, Max Poliakovski wrote: >> >> Elliot, >> >>> Grepping through System 9.2.2 reveals several compressed resources: >>> >>> Could you have used a search method that extracted the resources >>> automatically? >> >> >> No. I opened the resource fork in a Unix Hexeditor and searched for 0xA89F6572. >> According to ResEdit, both CDEF #62 and DRVR #2 are uncompressed in my german "System" file: the "compressed" checkbox isn't selected. >> Does ResEdit unpack resources in background when opening a file for edit? >> >> $ SimpleDeRez ~/System/..namedfork/rsrc -compcmt -ascii | grep -A3 _compressed | head -n9 >> data 'CDEF' (62, $61 /*sysheap, purgeable, _compressed*/) { >> $"A89F 6572 0012 0901 0000 0454 0003 03FF" /* ..er.......T.... */ >> $"0000 3F04 E56F FF44 8E70 7082 A2E0 0083" /* ..?..o.D.pp..... */ >> $"E2E0 0122 86E0 00E7 C0A2 0544 3EEF FF85" /* ...".......D>... */ >> -- >> data 'DRVR' (2, ".Print", $71 /*sysheap, purgeable, locked, _compressed*/) { >> $"A89F 6572 0012 0901 0000 019A 0003 03FF" /* ..er............ */ >> $"0000 2278 058F FC90 00A0 00B0 00C0 00D0" /* .."x............ */ >> $"18B9 41C9 A5B9 D080 0001 1149 5948 0008" /* ..A........IYH.. */ >> >>> >>> Here are some Donn resources, both compressed and extracted (manual >>> operation... ugh!). How will I identify the mysterious Mac OS 9 format? >> >> >> CDEF #62: >> $"A89F 6572 0012 0901 0000 0454 0003 03FF" >> DRVR #2: >> $"A89F 6572 0012 0901 0000 019A 0003 03FF" >> >> According to the defconID = 0x0003, both resources are compressed with that mysterious new method. >> >> A short peek into 'dcmp' #3 reveals a sophisticated compression method utilizing Huffman coding. It contains neither strings nor tables. Just a lot of code, much larger than Donn&Gregg stuff. I therefore suggest to postpone its RE until the Trampoline work is finished... >> >> BTW, many thanks for the Donn bits! I'll analyze them soon... >> >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >> >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Fri Dec 14 10:51:14 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 14 Dec 2018 03:51:14 +0100 Subject: [CDG5] Random questions In-Reply-To: <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: Hi guys, Good call on the Huffman code. Any chance it could be a well known algo? > Our mysterious compression method #3 looks like a variant of Deflate; it's based on the same principles: matching and reusing of already encoded data and Huffman coding. The decompression code includes two copy routines: one for copying data pointed by a backward distance and another one for copying bits directly from source to destination. They are controlled by various Huffman codes implemented as bit-crunching subroutines. There are 15 such subroutines corresponding to 15 various variable-length coding tables. As opposite to Deflate, OS9 novel Rez compression doesn't have any headers, just raw Huffman codes interleaved with plain-text data... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181214/fd6872c2/attachment.htm From jason.duerstock at gmail.com Fri Dec 14 11:12:43 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Thu, 13 Dec 2018 22:12:43 -0500 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: The "tome" format uses this "InstaCompOne" algorithm. It's possible they recycled that for resource compression as well. On Thu, Dec 13, 2018 at 9:52 PM Max Poliakovski wrote: > > Hi guys, > >> Good call on the Huffman code. Any chance it could be a well known algo? > > > Our mysterious compression method #3 looks like a variant of Deflate; it's based on the same principles: matching and reusing of already encoded data and Huffman coding. > > The decompression code includes two copy routines: one for copying data pointed by a backward distance and another one for copying bits directly from source to destination. They are controlled by various Huffman codes implemented as bit-crunching subroutines. There are 15 such subroutines corresponding to 15 various variable-length coding tables. > > As opposite to Deflate, OS9 novel Rez compression doesn't have any headers, just raw Huffman codes interleaved with plain-text data... > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Fri Dec 14 18:07:43 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 14 Dec 2018 11:07:43 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: > The "tome" format uses this "InstaCompOne" algorithm. It's possible > they recycled that for resource compression as well. > Interesting! Did you already locate the code responsible for "tome" decompression? I'd like to take a look at it... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181214/d1cad591/attachment.htm From jason.duerstock at gmail.com Sat Dec 15 00:36:03 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Fri, 14 Dec 2018 11:36:03 -0500 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: I unpacked the relevant Installer SDK resources and put them here: https://github.com/jduerstock/instacompone exfn.00f1 is the m68k version exfn.00f2 is the ppc version exfn.00f3 is the "fat" version Jason On Fri, Dec 14, 2018 at 5:12 AM Max Poliakovski wrote: > > >> The "tome" format uses this "InstaCompOne" algorithm. It's possible >> they recycled that for resource compression as well. > > > Interesting! Did you already locate the code responsible for "tome" decompression? I'd like to take a look at it... > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Sun Dec 16 09:37:39 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 16 Dec 2018 09:37:39 +0800 Subject: [CDG5] Attachments Message-ID: Hi all, Some of us (including me) have had trouble sending attachments to the list. I think it's best if in future we post external links instead. Not only do we enjoy the wonders of modern version control, but we also steer clear of any institutional policies on virus scanning and binary hosting. Elliot From maximumspatium at googlemail.com Sun Dec 16 09:45:46 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Sun, 16 Dec 2018 02:45:46 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: I'll repost my earlier message without attachments... Jason, how in the world did you discover that InstaComp stuff? You're absolutely right that the code in dcmp #3 has been derived from InstaComp - I was able to locate it within InstaCompOneTool application! The latter seems to offer more compression modes though. I don't know if you already own the InstaCompOne SDK from 1994. Anyway, it contains a MPW tool called InstaCompOneTool that can be directly invoked from the MPW shell for compressing, decompressing and viewing of InstaComp archives AND resources. It includes a guide in the DocViewer format as well (requires Apple DocViewer 1.1.1). That all is not only cool but also very handy because we got the appropriate compressor too. I'll dive into it tonight and see if I'll be able to figure out further details... P.S.: I already reversed that InstaComp algo. I'm writing a Python decompressor for it right now. The stuff has turned out to be easily understandable. Stay tuned. I'll commit that beast in a couple of days. That will be another Christmas gift for you, hehe! -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181216/efc03cbc/attachment.htm From jason.duerstock at gmail.com Sun Dec 16 10:14:59 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sat, 15 Dec 2018 21:14:59 -0500 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: Max, I found it when I was trying to figure out what they used to compress the installer "tome" files. I have the Installer SDK somewhere -- I was going to post a link to it once I found somewhere sane to stash it. If you already have your brain wrapped around InstaCompOne, I'll leave you to it. Jason On Sat, Dec 15, 2018 at 8:46 PM Max Poliakovski wrote: > > I'll repost my earlier message without attachments... > > Jason, > > how in the world did you discover that InstaComp stuff? You're absolutely right that the code in dcmp #3 has been derived from InstaComp - I was able to locate it within InstaCompOneTool application! The latter seems to offer more compression modes though. > > I don't know if you already own the InstaCompOne SDK from 1994. Anyway, it contains a MPW tool called InstaCompOneTool that can be directly invoked from the MPW shell for compressing, decompressing and viewing of InstaComp archives AND resources. It includes a guide in the DocViewer format as well (requires Apple DocViewer 1.1.1). > > That all is not only cool but also very handy because we got the appropriate compressor too. I'll dive into it tonight and see if I'll be able to figure out further details... > > P.S.: I already reversed that InstaComp algo. I'm writing a Python decompressor for it right now. The stuff has turned out to be easily understandable. Stay tuned. I'll commit that beast in a couple of days. That will be another Christmas gift for you, hehe! > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Tue Dec 18 08:21:08 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Tue, 18 Dec 2018 01:21:08 +0100 Subject: [CDG5] Random questions In-Reply-To: References: <1F162B0E-D1B2-4796-8774-2D79C8414A88@fastmail.com> <3CF74750-270E-4B95-A199-BB888CD80C0A@fastmail.com> <647054B8-2704-4DEB-8C60-A9C98C82F1EB@verizon.net> <990ED3BF-0AE0-4FA5-894A-CB8996F79C76@fastmail.com> <0C8D8AE7-4B72-4325-BB75-0400DE2B4455@fastmail.com> <20D12E8A-CEE8-4762-A59E-F6D64921D7CB@fastmail.com> Message-ID: Hi guys, I finally pushed an initial decompressor for the InstaCompOne algorithm to Github, see https://github.com/maximumspatium/ResDecompress/commit/6316cac2c5bbb88e23604be93493ac0f2bac2ff9 Although it currently decodes the gnld #0 resource just fine, the Huffman tables are still incomplete. Moreover, no optimization attempt has been ever made. It definitely need more testing. Anyway, I consider it a good starting point... Cheers Max -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181218/a74dfd1e/attachment.htm From elliotnunn at fastmail.com Sun Dec 23 13:58:22 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 23 Dec 2018 13:58:22 +0800 Subject: [CDG5] Linked Patches Message-ID: Anyone got a clue how the LinkPatch tool works? I am having trouble correlating the contents of System 7.1.2 with the Cube-E Sys.r file. (The idea is, like with the NewWorld ROM, to build something bootable.) From danielbj314 at verizon.net Sun Dec 23 22:27:11 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Sun, 23 Dec 2018 09:27:11 -0500 Subject: [CDG5] Linked Patches In-Reply-To: References: Message-ID: <008F0C37-F1E4-4A17-A094-3D79A12AA51A@verizon.net> > Anyone got a clue how the LinkPatch tool works? I am having trouble > correlating the contents of System 7.1.2 with the Cube-E Sys.r file. > (The idea is, like with the NewWorld ROM, to build something bootable.) I think that the linked patch files include a whole ton of macros from LinkedPatchMacros.a. The macros set up all kinds of stuff. At boot time, LinkedPatchLoader constructs jump tables for all the linked patch resources. I think. It was very confusing when I read it. I?m not even sure we?re talking about the same thing. Does my description make sense with what comments and source you are looking at? From evilmoo at gmail.com Sun Dec 23 21:57:12 2018 From: evilmoo at gmail.com (Jason Duerstock) Date: Sun, 23 Dec 2018 08:57:12 -0500 Subject: [CDG5] qemu fun Message-ID: f you want to see something a little brain-melting, run qemu against Mac OS 9.2 and turn on the assembly translation logging with "-d in_asm,out_asm". qemu-system-ppc -d in_asm,out_asm -L pc-bios -boot d -M mac99 -m 512 -hda macos922.qcow2 -cdrom macos_921_ppc.iso 2>asm.log This doesn't give an exact code path trace, but shows the sections that qemu translates. Getting the install CD booted and then telling it to shut down produces about 300MB worth of log. Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181223/0cf1cec9/attachment.htm From maximumspatium at googlemail.com Mon Dec 24 04:47:19 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Sun, 23 Dec 2018 21:47:19 +0100 Subject: [CDG5] qemu fun In-Reply-To: References: Message-ID: I call it "big data", hehe. Nice to have but it's no fun to scroll through... Am So., 23. Dez. 2018 um 15:43 Uhr schrieb Jason Duerstock < evilmoo at gmail.com>: > f you want to see something a little brain-melting, run qemu against Mac > OS 9.2 and turn on the assembly translation logging with "-d > in_asm,out_asm". > > qemu-system-ppc -d in_asm,out_asm -L pc-bios -boot d -M mac99 -m 512 -hda > macos922.qcow2 -cdrom macos_921_ppc.iso 2>asm.log > > This doesn't give an exact code path trace, but shows the sections that > qemu translates. Getting the install CD booted and then telling it to shut > down produces about 300MB worth of log. > > Jason > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181223/ed6207cc/attachment.htm From elliotnunn at fastmail.com Mon Dec 24 13:03:17 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 24 Dec 2018 13:03:17 +0800 Subject: [CDG5] Linked Patches In-Reply-To: <008F0C37-F1E4-4A17-A094-3D79A12AA51A@verizon.net> References: <008F0C37-F1E4-4A17-A094-3D79A12AA51A@verizon.net> Message-ID: <977BD254-7084-4F2E-876A-882D98556BE7@fastmail.com> That's going a bit over my head. But it turns out that the source tree has an object file and small assembly stub (all in LinkedPatches:) that when linked together produce a working LinkPatch. https://github.com/elliotnunn/LinkPatch And now to figure out what to do next. I assume that it postprocesses the System file, and maybe it leaves some hints around for the Installer. Time to read the error messages... > On 23 Dec 2018, at 10:27 pm, daniel B-J wrote: > >> Anyone got a clue how the LinkPatch tool works? I am having trouble >> correlating the contents of System 7.1.2 with the Cube-E Sys.r file. >> (The idea is, like with the NewWorld ROM, to build something bootable.) > > I think that the linked patch files include a whole ton of macros from LinkedPatchMacros.a. The macros set up all kinds of stuff. At boot time, LinkedPatchLoader constructs jump tables for all the linked patch resources. I think. It was very confusing when I read it. I?m not even sure we?re talking about the same thing. Does my description make sense with what comments and source you are looking at? > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Fri Dec 28 11:15:23 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 04:15:23 +0100 Subject: [CDG5] MPW code mysteries Message-ID: Fellow hackers, I'm currently trying to get MPW 3.5 (from EMPW) to generate code closely resembling the original Trampoline. While doing so, I immediately stumbled across two issues I cannot resolve. I must admit that my MPW skills are limited. I therefore hope to get some help from you. 1) Trampoline's startup code initializes the RTOC as follows: mflr r11 bl setup_rtoc DC.L 0x1001E8 setup_rtoc: mflr RTOC lwz RTOC, 0(RTOC) The word in the 3rd line contains the base address of TOC. The value shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to make PPCAsm/PPCLink to stuff the correct value there. That unfortunately doesn't work. PPCAsm says: ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. Make sure to add it to its csect's TOC entry. I don't understand this message. What do I need to put there in order to get the base address of TOC placed there? 2) Trampoline's main() initializes the pointer to OF client interface right at the beginning. The code of interest looks like that: lwz r9, CIPtr_TC(RTOC) lisori r3, 0xdeadbeef stw r5, 0(r9) stw r3, 4(r9) The pointer to the OF client dispatcher is in R5 (3rd argument of main). The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I cannot reproduce. I defined a correct function pointer like that: typedef long (*OfCiPtr)(CIArgs *args); static OfCIPtr gCIPtr; void main(int argc, int arg[], OfCiPtr ciPtr) { gCIPtr = ciPtr; } When I initialize that global pointer gCIPtr with value from ciPtr (3rd argument), only the function address is written while the TOC value remains untouched. The compiled code looks like that: lwz r9, CIPtr_TC(RTOC) stw r5, 0(r9) I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do you have any clue? -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/824fdcbd/attachment.htm From danielbj314 at verizon.net Fri Dec 28 11:34:31 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Thu, 27 Dec 2018 22:34:31 -0500 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: > I don't understand this message. What do I need to put there in order to get the base address of TOC placed there? Perhaps DC.L TOC[TC0] - Data_Start + Data_Base_Address? Data_Start is the label that ends up at 0x00100000 and Data_Base_Address is 0x00100000. Haven't tested it. > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do you have any clue? I'm guessing they did it in assembly. Or maybe there are some special macros in the Universal Interfaces for working with TVectors? Special casting trickery? You might be able to cast a TVector to an int array or a struct or something. Not sure. From maximumspatium at googlemail.com Fri Dec 28 12:00:29 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 05:00:29 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do > you have any clue? > I'm guessing they did it in assembly. Or maybe there are some special > macros in the Universal Interfaces for working with TVectors? Special > casting trickery? You might be able to cast a TVector to an int array or a > struct or something. Not sure. > Thank you for your suggestions! The following works: typedef struct TVector { void *ptr; long toc; } TVector; void main(int argc, int argv[], OfCiPtr ciPtr) { TVector *tvec; tvec = (TVector *)ciPtr; tvec->ptr = ciPtr; tvec->toc = 0xdeadbeef; } Anyway, that looks like an unnatural way to initialize transition vectors... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/952469e6/attachment.htm From maximumspatium at googlemail.com Fri Dec 28 12:02:05 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 05:02:05 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: Sorry, the correct code is: tvec = (TVector *)gCIPtr; Am Fr., 28. Dez. 2018 um 05:00 Uhr schrieb Max Poliakovski < maximumspatium at googlemail.com>: > > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do >> you have any clue? >> I'm guessing they did it in assembly. Or maybe there are some special >> macros in the Universal Interfaces for working with TVectors? Special >> casting trickery? You might be able to cast a TVector to an int array or a >> struct or something. Not sure. >> > > Thank you for your suggestions! The following works: > > typedef struct TVector { > void *ptr; > long toc; > } TVector; > > void main(int argc, int argv[], OfCiPtr ciPtr) > { > TVector *tvec; > > tvec = (TVector *)ciPtr; > tvec->ptr = ciPtr; > tvec->toc = 0xdeadbeef; > } > > Anyway, that looks like an unnatural way to initialize transition > vectors... > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/15959a09/attachment.htm From elliotnunn at fastmail.com Fri Dec 28 12:02:39 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Fri, 28 Dec 2018 12:02:39 +0800 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: I have to re-learn the AIX-style ABI every time I touch it, so bear with me. Apple's docs reveal the implementation detail that every C function pointer is actually a pointer to a TVector. (For those following along, a Transition Vector is an 8-byte structure consisting of a true code pointer and a pointer to the "Table of Contents" for that code. The TOC is analogous to an old-style 68k A5 world, except that a called function can assume that r2 already points to the TOC on entry. Therefore functions that call outside their own code fragment must do so through glue code that sets r2 to the callee's TOC, and each such bl instruction must be followed by an instruction to restore r2 to sanity.) So I'd just cast the function ptr to a TVector pointer and modify it that way: (*(unsigned long **)funcPtr)[0] = 0xdeadbeefUL; > On 28 Dec 2018, at 11:15 am, Max Poliakovski wrote: > > Fellow hackers, > > I'm currently trying to get MPW 3.5 (from EMPW) to generate code closely resembling the original Trampoline. > > While doing so, I immediately stumbled across two issues I cannot resolve. I must admit that my MPW skills are limited. I therefore hope to get some help from you. > > 1) Trampoline's startup code initializes the RTOC as follows: > > mflr r11 > bl setup_rtoc > DC.L 0x1001E8 > setup_rtoc: > mflr RTOC > lwz RTOC, 0(RTOC) > > The word in the 3rd line contains the base address of TOC. The value shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to make PPCAsm/PPCLink to stuff the correct value there. That unfortunately doesn't work. PPCAsm says: > > ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. Make sure to add it to its csect's TOC entry. > > I don't understand this message. What do I need to put there in order to get the base address of TOC placed there? > > 2) Trampoline's main() initializes the pointer to OF client interface right at the beginning. The code of interest looks like that: > > lwz r9, CIPtr_TC(RTOC) > lisori r3, 0xdeadbeef > stw r5, 0(r9) > stw r3, 4(r9) > > The pointer to the OF client dispatcher is in R5 (3rd argument of main). The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I cannot reproduce. I defined a correct function pointer like that: > > typedef long (*OfCiPtr)(CIArgs *args); > static OfCIPtr gCIPtr; > > void main(int argc, int arg[], OfCiPtr ciPtr) > { > gCIPtr = ciPtr; > } > > When I initialize that global pointer gCIPtr with value from ciPtr (3rd argument), only the function address is written while the TOC value remains untouched. The compiled code looks like that: > > lwz r9, CIPtr_TC(RTOC) > stw r5, 0(r9) > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do you have any clue? > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Fri Dec 28 12:04:57 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Fri, 28 Dec 2018 12:04:57 +0800 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: Beaten to it! > On 28 Dec 2018, at 12:02 pm, Elliot Nunn wrote: > > I have to re-learn the AIX-style ABI every time I touch it, so bear with > me. > > Apple's docs reveal the implementation detail that every C function > pointer is actually a pointer to a TVector. > > (For those following along, a Transition Vector is an 8-byte structure > consisting of a true code pointer and a pointer to the "Table of > Contents" for that code. The TOC is analogous to an old-style 68k A5 > world, except that a called function can assume that r2 already points > to the TOC on entry. Therefore functions that call outside their own > code fragment must do so through glue code that sets r2 to the callee's > TOC, and each such bl instruction must be followed by an instruction to > restore r2 to sanity.) > > So I'd just cast the function ptr to a TVector pointer and modify it > that way: > > (*(unsigned long **)funcPtr)[0] = 0xdeadbeefUL; > >> On 28 Dec 2018, at 11:15 am, Max Poliakovski wrote: >> >> Fellow hackers, >> >> I'm currently trying to get MPW 3.5 (from EMPW) to generate code closely resembling the original Trampoline. >> >> While doing so, I immediately stumbled across two issues I cannot resolve. I must admit that my MPW skills are limited. I therefore hope to get some help from you. >> >> 1) Trampoline's startup code initializes the RTOC as follows: >> >> mflr r11 >> bl setup_rtoc >> DC.L 0x1001E8 >> setup_rtoc: >> mflr RTOC >> lwz RTOC, 0(RTOC) >> >> The word in the 3rd line contains the base address of TOC. The value shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to make PPCAsm/PPCLink to stuff the correct value there. That unfortunately doesn't work. PPCAsm says: >> >> ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. Make sure to add it to its csect's TOC entry. >> >> I don't understand this message. What do I need to put there in order to get the base address of TOC placed there? >> >> 2) Trampoline's main() initializes the pointer to OF client interface right at the beginning. The code of interest looks like that: >> >> lwz r9, CIPtr_TC(RTOC) >> lisori r3, 0xdeadbeef >> stw r5, 0(r9) >> stw r3, 4(r9) >> >> The pointer to the OF client dispatcher is in R5 (3rd argument of main). The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I cannot reproduce. I defined a correct function pointer like that: >> >> typedef long (*OfCiPtr)(CIArgs *args); >> static OfCIPtr gCIPtr; >> >> void main(int argc, int arg[], OfCiPtr ciPtr) >> { >> gCIPtr = ciPtr; >> } >> >> When I initialize that global pointer gCIPtr with value from ciPtr (3rd argument), only the function address is written while the TOC value remains untouched. The compiled code looks like that: >> >> lwz r9, CIPtr_TC(RTOC) >> stw r5, 0(r9) >> >> I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do you have any clue? >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Fri Dec 28 12:07:44 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 05:07:44 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: Interesting suggestion. Many thanks! Do you surely mean (*(unsigned long **)funcPtr)[1] = 0xdeadbeefUL; ? Am Fr., 28. Dez. 2018 um 05:03 Uhr schrieb Elliot Nunn < elliotnunn at fastmail.com>: > I have to re-learn the AIX-style ABI every time I touch it, so bear with > me. > > Apple's docs reveal the implementation detail that every C function > pointer is actually a pointer to a TVector. > > (For those following along, a Transition Vector is an 8-byte structure > consisting of a true code pointer and a pointer to the "Table of > Contents" for that code. The TOC is analogous to an old-style 68k A5 > world, except that a called function can assume that r2 already points > to the TOC on entry. Therefore functions that call outside their own > code fragment must do so through glue code that sets r2 to the callee's > TOC, and each such bl instruction must be followed by an instruction to > restore r2 to sanity.) > > So I'd just cast the function ptr to a TVector pointer and modify it > that way: > > (*(unsigned long **)funcPtr)[0] = 0xdeadbeefUL; > > > On 28 Dec 2018, at 11:15 am, Max Poliakovski < > maximumspatium at googlemail.com> wrote: > > > > Fellow hackers, > > > > I'm currently trying to get MPW 3.5 (from EMPW) to generate code closely > resembling the original Trampoline. > > > > While doing so, I immediately stumbled across two issues I cannot > resolve. I must admit that my MPW skills are limited. I therefore hope to > get some help from you. > > > > 1) Trampoline's startup code initializes the RTOC as follows: > > > > mflr r11 > > bl setup_rtoc > > DC.L 0x1001E8 > > setup_rtoc: > > mflr RTOC > > lwz RTOC, 0(RTOC) > > > > The word in the 3rd line contains the base address of TOC. The value > shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to > make PPCAsm/PPCLink to stuff the correct value there. That unfortunately > doesn't work. PPCAsm says: > > > > ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. > Make sure to add it to its csect's TOC entry. > > > > I don't understand this message. What do I need to put there in order to > get the base address of TOC placed there? > > > > 2) Trampoline's main() initializes the pointer to OF client interface > right at the beginning. The code of interest looks like that: > > > > lwz r9, CIPtr_TC(RTOC) > > lisori r3, 0xdeadbeef > > stw r5, 0(r9) > > stw r3, 4(r9) > > > > The pointer to the OF client dispatcher is in R5 (3rd argument of main). > The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I cannot > reproduce. I defined a correct function pointer like that: > > > > typedef long (*OfCiPtr)(CIArgs *args); > > static OfCIPtr gCIPtr; > > > > void main(int argc, int arg[], OfCiPtr ciPtr) > > { > > gCIPtr = ciPtr; > > } > > > > When I initialize that global pointer gCIPtr with value from ciPtr (3rd > argument), only the function address is written while the TOC value remains > untouched. The compiled code looks like that: > > > > lwz r9, CIPtr_TC(RTOC) > > stw r5, 0(r9) > > > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do > you have any clue? > > _______________________________________________ > > cdg5 mailing list > > cdg5 at ucc.asn.au > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/b686c7cb/attachment.htm From elliotnunn at fastmail.com Fri Dec 28 12:45:01 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Fri, 28 Dec 2018 12:45:01 +0800 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: <81788E02-C045-4A0F-8494-C330CF265C5D@fastmail.com> Yes, my mistake. You want to change the TOC ptr, not the code ptr. > On 28 Dec 2018, at 12:07 pm, Max Poliakovski wrote: > > Interesting suggestion. Many thanks! > > Do you surely mean > > (*(unsigned long **)funcPtr)[1] = 0xdeadbeefUL; > > ? > > Am Fr., 28. Dez. 2018 um 05:03 Uhr schrieb Elliot Nunn : > I have to re-learn the AIX-style ABI every time I touch it, so bear with > me. > > Apple's docs reveal the implementation detail that every C function > pointer is actually a pointer to a TVector. > > (For those following along, a Transition Vector is an 8-byte structure > consisting of a true code pointer and a pointer to the "Table of > Contents" for that code. The TOC is analogous to an old-style 68k A5 > world, except that a called function can assume that r2 already points > to the TOC on entry. Therefore functions that call outside their own > code fragment must do so through glue code that sets r2 to the callee's > TOC, and each such bl instruction must be followed by an instruction to > restore r2 to sanity.) > > So I'd just cast the function ptr to a TVector pointer and modify it > that way: > > (*(unsigned long **)funcPtr)[0] = 0xdeadbeefUL; > > > On 28 Dec 2018, at 11:15 am, Max Poliakovski wrote: > > > > Fellow hackers, > > > > I'm currently trying to get MPW 3.5 (from EMPW) to generate code closely resembling the original Trampoline. > > > > While doing so, I immediately stumbled across two issues I cannot resolve. I must admit that my MPW skills are limited. I therefore hope to get some help from you. > > > > 1) Trampoline's startup code initializes the RTOC as follows: > > > > mflr r11 > > bl setup_rtoc > > DC.L 0x1001E8 > > setup_rtoc: > > mflr RTOC > > lwz RTOC, 0(RTOC) > > > > The word in the 3rd line contains the base address of TOC. The value shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to make PPCAsm/PPCLink to stuff the correct value there. That unfortunately doesn't work. PPCAsm says: > > > > ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. Make sure to add it to its csect's TOC entry. > > > > I don't understand this message. What do I need to put there in order to get the base address of TOC placed there? > > > > 2) Trampoline's main() initializes the pointer to OF client interface right at the beginning. The code of interest looks like that: > > > > lwz r9, CIPtr_TC(RTOC) > > lisori r3, 0xdeadbeef > > stw r5, 0(r9) > > stw r3, 4(r9) > > > > The pointer to the OF client dispatcher is in R5 (3rd argument of main). The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I cannot reproduce. I defined a correct function pointer like that: > > > > typedef long (*OfCiPtr)(CIArgs *args); > > static OfCIPtr gCIPtr; > > > > void main(int argc, int arg[], OfCiPtr ciPtr) > > { > > gCIPtr = ciPtr; > > } > > > > When I initialize that global pointer gCIPtr with value from ciPtr (3rd argument), only the function address is written while the TOC value remains untouched. The compiled code looks like that: > > > > lwz r9, CIPtr_TC(RTOC) > > stw r5, 0(r9) > > > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? Do you have any clue? > > _______________________________________________ > > cdg5 mailing list > > cdg5 at ucc.asn.au > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From maximumspatium at googlemail.com Fri Dec 28 12:48:57 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 05:48:57 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: <81788E02-C045-4A0F-8494-C330CF265C5D@fastmail.com> References: <81788E02-C045-4A0F-8494-C330CF265C5D@fastmail.com> Message-ID: I finally figured out how to do that without defining TVector: gCIPtr = ciPtr; ((unsigned long *)&gCIPtr)[1] = 0xdeadbeefUL; Daniel & Elliot, thank you for your suggestions! Am Fr., 28. Dez. 2018 um 05:45 Uhr schrieb Elliot Nunn < elliotnunn at fastmail.com>: > Yes, my mistake. You want to change the TOC ptr, not the code ptr. > > > On 28 Dec 2018, at 12:07 pm, Max Poliakovski < > maximumspatium at googlemail.com> wrote: > > > > Interesting suggestion. Many thanks! > > > > Do you surely mean > > > > (*(unsigned long **)funcPtr)[1] = 0xdeadbeefUL; > > > > ? > > > > Am Fr., 28. Dez. 2018 um 05:03 Uhr schrieb Elliot Nunn < > elliotnunn at fastmail.com>: > > I have to re-learn the AIX-style ABI every time I touch it, so bear with > > me. > > > > Apple's docs reveal the implementation detail that every C function > > pointer is actually a pointer to a TVector. > > > > (For those following along, a Transition Vector is an 8-byte structure > > consisting of a true code pointer and a pointer to the "Table of > > Contents" for that code. The TOC is analogous to an old-style 68k A5 > > world, except that a called function can assume that r2 already points > > to the TOC on entry. Therefore functions that call outside their own > > code fragment must do so through glue code that sets r2 to the callee's > > TOC, and each such bl instruction must be followed by an instruction to > > restore r2 to sanity.) > > > > So I'd just cast the function ptr to a TVector pointer and modify it > > that way: > > > > (*(unsigned long **)funcPtr)[0] = 0xdeadbeefUL; > > > > > On 28 Dec 2018, at 11:15 am, Max Poliakovski < > maximumspatium at googlemail.com> wrote: > > > > > > Fellow hackers, > > > > > > I'm currently trying to get MPW 3.5 (from EMPW) to generate code > closely resembling the original Trampoline. > > > > > > While doing so, I immediately stumbled across two issues I cannot > resolve. I must admit that my MPW skills are limited. I therefore hope to > get some help from you. > > > > > > 1) Trampoline's startup code initializes the RTOC as follows: > > > > > > mflr r11 > > > bl setup_rtoc > > > DC.L 0x1001E8 > > > setup_rtoc: > > > mflr RTOC > > > lwz RTOC, 0(RTOC) > > > > > > The word in the 3rd line contains the base address of TOC. The value > shown above is from Apple's binary. I replaced it with DC.L TOC[TC0] to > make PPCAsm/PPCLink to stuff the correct value there. That unfortunately > doesn't work. PPCAsm says: > > > > > > ### Warning 3171 ### Treating 'TOC[TC0]' as a csect relative offset. > Make sure to add it to its csect's TOC entry. > > > > > > I don't understand this message. What do I need to put there in order > to get the base address of TOC placed there? > > > > > > 2) Trampoline's main() initializes the pointer to OF client interface > right at the beginning. The code of interest looks like that: > > > > > > lwz r9, CIPtr_TC(RTOC) > > > lisori r3, 0xdeadbeef > > > stw r5, 0(r9) > > > stw r3, 4(r9) > > > > > > The pointer to the OF client dispatcher is in R5 (3rd argument of > main). The TOC pointer is initialized to 0xDEADBEEF. That's exactly what I > cannot reproduce. I defined a correct function pointer like that: > > > > > > typedef long (*OfCiPtr)(CIArgs *args); > > > static OfCIPtr gCIPtr; > > > > > > void main(int argc, int arg[], OfCiPtr ciPtr) > > > { > > > gCIPtr = ciPtr; > > > } > > > > > > When I initialize that global pointer gCIPtr with value from ciPtr > (3rd argument), only the function address is written while the TOC value > remains untouched. The compiled code looks like that: > > > > > > lwz r9, CIPtr_TC(RTOC) > > > stw r5, 0(r9) > > > > > > I wonder how Apple guys managed to put 0xDEADBEEF into TVector's TOC? > Do you have any clue? > > > _______________________________________________ > > > cdg5 mailing list > > > cdg5 at ucc.asn.au > > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > > > _______________________________________________ > > cdg5 mailing list > > cdg5 at ucc.asn.au > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > > cdg5 mailing list > > cdg5 at ucc.asn.au > > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/f131a3c5/attachment.htm From maximumspatium at googlemail.com Sat Dec 29 05:02:06 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 22:02:06 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: > > I don't understand this message. What do I need to put there in order to > get the base address of TOC placed there? > Perhaps DC.L TOC[TC0] - Data_Start + Data_Base_Address? Data_Start is the > label that ends up at 0x00100000 and Data_Base_Address is 0x00100000. > Haven't tested it. PPCAsm says that neither Data_Start nor Data_Base_Address are known so I'm stuck for now. I copied the TOC definition DC.L TOC[TC0] from transition vectors. It seems to emit the TOC base address just fine from inside a CSECT [DS] section. But the executable code is wrapped in a CSECT [PR] section so it doesn't work there as expected. TOC[TC0] seems to return an offset but I need the absolute address of TOC. Where can I ask for help on this? Code examples are difficult to find, PPCAsm doc is brief and I'm bad at reading between the lines... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/f8b0016e/attachment.htm From danielbj314 at verizon.net Sat Dec 29 05:35:10 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Fri, 28 Dec 2018 16:35:10 -0500 Subject: [CDG5] MPW code mysteries In-Reply-To: References: Message-ID: <4FC6877F-721A-4340-8E50-F01F9014CC52@verizon.net> > PPCAsm says that neither Data_Start nor Data_Base_Address are known so I'm stuck for now. I guess I wasn?t clear enough. Those symbols don?t exist. I meant that you should stick a symbol at the start of the data section and use that in your calculations. Call it whatever you want. Data_Base_Address should probably just be the constant 0x00100000. DC.L TOC[TC0] - + 0x00100000 The ?? are just part of the insert here message, not actual PPCASM syntax. From maximumspatium at googlemail.com Sat Dec 29 05:48:24 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 22:48:24 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: <4FC6877F-721A-4340-8E50-F01F9014CC52@verizon.net> References: <4FC6877F-721A-4340-8E50-F01F9014CC52@verizon.net> Message-ID: Daniel, thanks a lot! I could try that out although I'm afraid it won't work as expected. To my understanding, every object module produced by whatever tool contains its own TOC. PPCLink will finally merge all TOCs from all objects in a way we have no control over. The question is how to put a symbol at the start of the data section to ensure that it will end up at the start of the data section of the linked object? Am Fr., 28. Dez. 2018 um 22:35 Uhr schrieb daniel B-J < danielbj314 at verizon.net>: > > PPCAsm says that neither Data_Start nor Data_Base_Address are known so > I'm stuck for now. > > I guess I wasn?t clear enough. Those symbols don?t exist. I meant that you > should stick a symbol at the start of the data section and use that in your > calculations. Call it whatever you want. Data_Base_Address should probably > just be the constant 0x00100000. > > DC.L TOC[TC0] - here> + 0x00100000 > > The ?? are just part of the insert here message, not actual > PPCASM syntax. > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/3be102f9/attachment.htm From danielbj314 at verizon.net Sat Dec 29 06:07:43 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Fri, 28 Dec 2018 17:07:43 -0500 Subject: [CDG5] MPW code mysteries In-Reply-To: References: <4FC6877F-721A-4340-8E50-F01F9014CC52@verizon.net> Message-ID: <9998FA75-EC22-4B85-B6B7-979AAED236C7@verizon.net> > On Dec 28, 2018, at 4:48 PM, Max Poliakovski wrote: > > Daniel, thanks a lot! I could try that out although I'm afraid it won't work as expected. To my understanding, every object module produced by whatever tool contains its own TOC. PPCLink will finally merge all TOCs from all objects in a way we have no control over. The question is how to put a symbol at the start of the data section to ensure that it will end up at the start of the data section of the linked object? No clue. I note that Apple?s entire point of putting that pointer to the TOC there is because ELF files don?t initialize the TOC for you. If you were to compile for XCOFF, you could just leave that code snippet out entirely. Maybe the TOC was put there by some late-stage tool that converts the XCOFF to ELF? From maximumspatium at googlemail.com Sat Dec 29 06:26:42 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Fri, 28 Dec 2018 23:26:42 +0100 Subject: [CDG5] MPW code mysteries In-Reply-To: <9998FA75-EC22-4B85-B6B7-979AAED236C7@verizon.net> References: <4FC6877F-721A-4340-8E50-F01F9014CC52@verizon.net> <9998FA75-EC22-4B85-B6B7-979AAED236C7@verizon.net> Message-ID: > No clue. I note that Apple?s entire point of putting that pointer to the > TOC there is because ELF files don?t initialize the TOC for you. If you > were to compile for XCOFF, you could just leave that code snippet out > entirely. Unfortunately no. Every object module has to define an entry point. If no custom entry point was specified, PPCLink will link to "__start" from StdCRuntime.o which, in turn, depends on InterfaceLib etc. In other words, tons of code Trampoline doesn't need. Trampoline's start code initializes the stack pointer as well. Stack area is located in the BSS section and is 65536 bytes big. Referencing this stack area in the assembly was easy. > Maybe the TOC was put there by some late-stage tool that converts the > XCOFF to ELF? > This is surely possible and easy doable. I'll reconsider this solution later when writing a XCOFF->ELF converter... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181228/0f9fea95/attachment.htm From elliotnunn at fastmail.com Sun Dec 30 10:51:09 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 30 Dec 2018 10:51:09 +0800 Subject: [CDG5] Process Manager Message-ID: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> Like Max, I had a bit of a code-generation problem last night. Thought I'd share it for your entertainment. As a big part of getting a System build to work, I am recreating the makefiles for the Process Manager. The Process Mgr has a very novel runtime (it actually gets loaded from 'scod' resources by the Segment Loader). With not many clues around about how to build it, I decided that I should take a deep dive at this point and get it round-tripping. System 7.1 seems to be a good fit. For context, the build is currently producing about 1/2 to 2/3 of a System file, albeit of dubious quality. Here is the function in question. /* CheckUnitTableEntry. See if the UNITTABLE entry that this driver would occupy * is acceptable. For now, just checks whether entry is already in use. */ OSErr CheckUnitTableEntry(short resourceID) { register DCtlPtr dCtlPtr; DCtlHandle dCtlHdl; OSErr result; result = noErr; dCtlHdl = (DCtlHandle) UNITTABLE[resourceID]; if ( (dCtlHdl != nil) && ((dCtlPtr = *dCtlHdl) != nil) && ((dCtlPtr->dCtlFlags & DOpened) != 0) ) result = portInUse; return(result); } UNITTABLE is defined as (*(Handle **)(0x11C)) The unit table access was compiling to "EXT.L D0; MOVE.L $11C,A0; MOVE.L $0(A0,D0.L*4),A3" when I wanted "EXT.L D0; MOVE.L $11C,A0; ASL #2,D0; MOVE.L $0(A0,D0.L),A3". I fought with this until 1 am, changing headers, types and compiler versions to no avail. I found that the same motif appears in two or three places in that segment, and the ASL persists up to System 7.1.2, after which it was refactored out of existence. This morning I was about to email the list for help, when in a testament to the "starting really hard at a problem" methodology, the answer came to me. "MOVE.L $0(A0,D0.L*4),A3" is a very fancy-looking addressing mode, I thought. It wouldn't work on a vanilla M68k... Oh! System 7.1 *does* run on a 68000, because it supports the Plus and Classic. (I should know, because I grew up using it on a Classic.) The C compiler was being run with the -mc68020 argument because I copied the makefile from the ROM build. Now fixed! From danielbj314 at verizon.net Sun Dec 30 11:00:35 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sat, 29 Dec 2018 22:00:35 -0500 Subject: [CDG5] Process Manager In-Reply-To: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> Message-ID: > As a big part of getting a System build to work, I am recreating the > makefiles for the Process Manager. The Process Mgr has a very novel > runtime (it actually gets loaded from 'scod' resources by the Segment > Loader). With not many clues around about how to build it, I decided > that I should take a deep dive at this point and get it round-tripping. > System 7.1 seems to be a good fit. Sounds wacky and headachy. Either a very fun challenge or a very not fun challenge. Not sure which. From jason.duerstock at gmail.com Sun Dec 30 11:31:51 2018 From: jason.duerstock at gmail.com (Jason Duerstock) Date: Sat, 29 Dec 2018 22:31:51 -0500 Subject: [CDG5] Process Manager In-Reply-To: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> Message-ID: Man, you should've asked. "Check the compiler flags" would've been the first thing I thought of. On Sat, Dec 29, 2018 at 9:51 PM Elliot Nunn wrote: > Like Max, I had a bit of a code-generation problem last night. Thought > I'd share it for your entertainment. > > As a big part of getting a System build to work, I am recreating the > makefiles for the Process Manager. The Process Mgr has a very novel > runtime (it actually gets loaded from 'scod' resources by the Segment > Loader). With not many clues around about how to build it, I decided > that I should take a deep dive at this point and get it round-tripping. > System 7.1 seems to be a good fit. > > For context, the build is currently producing about 1/2 to 2/3 of a > System file, albeit of dubious quality. > > Here is the function in question. > > /* CheckUnitTableEntry. See if the UNITTABLE entry that this > driver would occupy > * is acceptable. For now, just checks whether entry is already > in use. > */ > OSErr > CheckUnitTableEntry(short resourceID) > { > register DCtlPtr dCtlPtr; > DCtlHandle dCtlHdl; > OSErr result; > > result = noErr; > dCtlHdl = (DCtlHandle) UNITTABLE[resourceID]; > if ( (dCtlHdl != nil) && ((dCtlPtr = *dCtlHdl) != nil) && > ((dCtlPtr->dCtlFlags & DOpened) != 0) ) > result = portInUse; > > return(result); > } > > UNITTABLE is defined as (*(Handle **)(0x11C)) > > The unit table access was compiling to "EXT.L D0; MOVE.L $11C,A0; MOVE.L > $0(A0,D0.L*4),A3" when I wanted "EXT.L D0; MOVE.L $11C,A0; ASL #2,D0; > MOVE.L $0(A0,D0.L),A3". I fought with this until 1 am, changing headers, > types and compiler versions to no avail. I found that the same motif > appears in two or three places in that segment, and the ASL persists up > to System 7.1.2, after which it was refactored out of existence. > > This morning I was about to email the list for help, when in a testament > to the "starting really hard at a problem" methodology, the answer came > to me. "MOVE.L $0(A0,D0.L*4),A3" is a very fancy-looking addressing > mode, I thought. It wouldn't work on a vanilla M68k... Oh! System 7.1 > *does* run on a 68000, because it supports the Plus and Classic. (I > should know, because I grew up using it on a Classic.) The C compiler > was being run with the -mc68020 argument because I copied the makefile > from the ROM build. Now fixed! > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181229/7fe012a0/attachment.htm From elliotnunn at fastmail.com Sun Dec 30 11:34:15 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 30 Dec 2018 11:34:15 +0800 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> Message-ID: <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Didn't learn my lesson, either. I just spent 30 minutes learning that '-opt full' *might* somehow affect code generation. > On 30 Dec 2018, at 11:31 am, Jason Duerstock wrote: > > Man, you should've asked. "Check the compiler flags" would've been the first thing I thought of. > > On Sat, Dec 29, 2018 at 9:51 PM Elliot Nunn wrote: > Like Max, I had a bit of a code-generation problem last night. Thought > I'd share it for your entertainment. > > As a big part of getting a System build to work, I am recreating the > makefiles for the Process Manager. The Process Mgr has a very novel > runtime (it actually gets loaded from 'scod' resources by the Segment > Loader). With not many clues around about how to build it, I decided > that I should take a deep dive at this point and get it round-tripping. > System 7.1 seems to be a good fit. > > For context, the build is currently producing about 1/2 to 2/3 of a > System file, albeit of dubious quality. > > Here is the function in question. > > /* CheckUnitTableEntry. See if the UNITTABLE entry that this driver would occupy > * is acceptable. For now, just checks whether entry is already in use. > */ > OSErr > CheckUnitTableEntry(short resourceID) > { > register DCtlPtr dCtlPtr; > DCtlHandle dCtlHdl; > OSErr result; > > result = noErr; > dCtlHdl = (DCtlHandle) UNITTABLE[resourceID]; > if ( (dCtlHdl != nil) && ((dCtlPtr = *dCtlHdl) != nil) && ((dCtlPtr->dCtlFlags & DOpened) != 0) ) > result = portInUse; > > return(result); > } > > UNITTABLE is defined as (*(Handle **)(0x11C)) > > The unit table access was compiling to "EXT.L D0; MOVE.L $11C,A0; MOVE.L > $0(A0,D0.L*4),A3" when I wanted "EXT.L D0; MOVE.L $11C,A0; ASL #2,D0; > MOVE.L $0(A0,D0.L),A3". I fought with this until 1 am, changing headers, > types and compiler versions to no avail. I found that the same motif > appears in two or three places in that segment, and the ASL persists up > to System 7.1.2, after which it was refactored out of existence. > > This morning I was about to email the list for help, when in a testament > to the "starting really hard at a problem" methodology, the answer came > to me. "MOVE.L $0(A0,D0.L*4),A3" is a very fancy-looking addressing > mode, I thought. It wouldn't work on a vanilla M68k... Oh! System 7.1 > *does* run on a 68000, because it supports the Plus and Classic. (I > should know, because I grew up using it on a Classic.) The C compiler > was being run with the -mc68020 argument because I copied the makefile > from the ROM build. Now fixed! > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Sun Dec 30 16:52:39 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Sun, 30 Dec 2018 16:52:39 +0800 Subject: [CDG5] Process Manager In-Reply-To: <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Message-ID: https://github.com/elliotnunn/CubeE/tree/processmgr-snapshot I can build the Process Manager! It *almost* round-trips, except that the linker puts the segments in the wrong order. Once that is fixed I will be able to clear up any lurking compilation problems. The commit message is the command to pipe into BuildCubeE. The commit is very messy, but I wanted to show this part off. > On 30 Dec 2018, at 11:34 am, Elliot Nunn wrote: > > Didn't learn my lesson, either. I just spent 30 minutes learning that > '-opt full' *might* somehow affect code generation. > >> On 30 Dec 2018, at 11:31 am, Jason Duerstock wrote: >> >> Man, you should've asked. "Check the compiler flags" would've been the first thing I thought of. >> >> On Sat, Dec 29, 2018 at 9:51 PM Elliot Nunn wrote: >> Like Max, I had a bit of a code-generation problem last night. Thought >> I'd share it for your entertainment. >> >> As a big part of getting a System build to work, I am recreating the >> makefiles for the Process Manager. The Process Mgr has a very novel >> runtime (it actually gets loaded from 'scod' resources by the Segment >> Loader). With not many clues around about how to build it, I decided >> that I should take a deep dive at this point and get it round-tripping. >> System 7.1 seems to be a good fit. >> >> For context, the build is currently producing about 1/2 to 2/3 of a >> System file, albeit of dubious quality. >> >> Here is the function in question. >> >> /* CheckUnitTableEntry. See if the UNITTABLE entry that this driver would occupy >> * is acceptable. For now, just checks whether entry is already in use. >> */ >> OSErr >> CheckUnitTableEntry(short resourceID) >> { >> register DCtlPtr dCtlPtr; >> DCtlHandle dCtlHdl; >> OSErr result; >> >> result = noErr; >> dCtlHdl = (DCtlHandle) UNITTABLE[resourceID]; >> if ( (dCtlHdl != nil) && ((dCtlPtr = *dCtlHdl) != nil) && ((dCtlPtr->dCtlFlags & DOpened) != 0) ) >> result = portInUse; >> >> return(result); >> } >> >> UNITTABLE is defined as (*(Handle **)(0x11C)) >> >> The unit table access was compiling to "EXT.L D0; MOVE.L $11C,A0; MOVE.L >> $0(A0,D0.L*4),A3" when I wanted "EXT.L D0; MOVE.L $11C,A0; ASL #2,D0; >> MOVE.L $0(A0,D0.L),A3". I fought with this until 1 am, changing headers, >> types and compiler versions to no avail. I found that the same motif >> appears in two or three places in that segment, and the ASL persists up >> to System 7.1.2, after which it was refactored out of existence. >> >> This morning I was about to email the list for help, when in a testament >> to the "starting really hard at a problem" methodology, the answer came >> to me. "MOVE.L $0(A0,D0.L*4),A3" is a very fancy-looking addressing >> mode, I thought. It wouldn't work on a vanilla M68k... Oh! System 7.1 >> *does* run on a 68000, because it supports the Plus and Classic. (I >> should know, because I grew up using it on a Classic.) The C compiler >> was being run with the -mc68020 argument because I copied the makefile >> from the ROM build. Now fixed! >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Mon Dec 31 01:04:29 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 01:04:29 +0800 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Message-ID: *Almost* round-tripping. All procedures and globals are placed correctly, and there are only one or two compiler niggles left. But a very small number of jump table entries are getting transposed, resulting in a few red speckles in vbindiff. > On 30 Dec 2018, at 4:52 pm, Elliot Nunn wrote: > > https://github.com/elliotnunn/CubeE/tree/processmgr-snapshot > > I can build the Process Manager! It *almost* round-trips, except that > the linker puts the segments in the wrong order. Once that is fixed I > will be able to clear up any lurking compilation problems. > > The commit message is the command to pipe into BuildCubeE. The commit is > very messy, but I wanted to show this part off. > >> On 30 Dec 2018, at 11:34 am, Elliot Nunn wrote: >> >> Didn't learn my lesson, either. I just spent 30 minutes learning that >> '-opt full' *might* somehow affect code generation. >> >>> On 30 Dec 2018, at 11:31 am, Jason Duerstock wrote: >>> >>> Man, you should've asked. "Check the compiler flags" would've been the first thing I thought of. >>> >>> On Sat, Dec 29, 2018 at 9:51 PM Elliot Nunn wrote: >>> Like Max, I had a bit of a code-generation problem last night. Thought >>> I'd share it for your entertainment. >>> >>> As a big part of getting a System build to work, I am recreating the >>> makefiles for the Process Manager. The Process Mgr has a very novel >>> runtime (it actually gets loaded from 'scod' resources by the Segment >>> Loader). With not many clues around about how to build it, I decided >>> that I should take a deep dive at this point and get it round-tripping. >>> System 7.1 seems to be a good fit. >>> >>> For context, the build is currently producing about 1/2 to 2/3 of a >>> System file, albeit of dubious quality. >>> >>> Here is the function in question. >>> >>> /* CheckUnitTableEntry. See if the UNITTABLE entry that this driver would occupy >>> * is acceptable. For now, just checks whether entry is already in use. >>> */ >>> OSErr >>> CheckUnitTableEntry(short resourceID) >>> { >>> register DCtlPtr dCtlPtr; >>> DCtlHandle dCtlHdl; >>> OSErr result; >>> >>> result = noErr; >>> dCtlHdl = (DCtlHandle) UNITTABLE[resourceID]; >>> if ( (dCtlHdl != nil) && ((dCtlPtr = *dCtlHdl) != nil) && ((dCtlPtr->dCtlFlags & DOpened) != 0) ) >>> result = portInUse; >>> >>> return(result); >>> } >>> >>> UNITTABLE is defined as (*(Handle **)(0x11C)) >>> >>> The unit table access was compiling to "EXT.L D0; MOVE.L $11C,A0; MOVE.L >>> $0(A0,D0.L*4),A3" when I wanted "EXT.L D0; MOVE.L $11C,A0; ASL #2,D0; >>> MOVE.L $0(A0,D0.L),A3". I fought with this until 1 am, changing headers, >>> types and compiler versions to no avail. I found that the same motif >>> appears in two or three places in that segment, and the ASL persists up >>> to System 7.1.2, after which it was refactored out of existence. >>> >>> This morning I was about to email the list for help, when in a testament >>> to the "starting really hard at a problem" methodology, the answer came >>> to me. "MOVE.L $0(A0,D0.L*4),A3" is a very fancy-looking addressing >>> mode, I thought. It wouldn't work on a vanilla M68k... Oh! System 7.1 >>> *does* run on a 68000, because it supports the Plus and Classic. (I >>> should know, because I grew up using it on a Classic.) The C compiler >>> was being run with the -mc68020 argument because I copied the makefile >>> from the ROM build. Now fixed! >>> _______________________________________________ >>> cdg5 mailing list >>> cdg5 at ucc.asn.au >>> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >>> _______________________________________________ >>> cdg5 mailing list >>> cdg5 at ucc.asn.au >>> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 >> >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From danielbj314 at verizon.net Mon Dec 31 01:12:53 2018 From: danielbj314 at verizon.net (daniel B-J) Date: Sun, 30 Dec 2018 12:12:53 -0500 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Message-ID: > *Almost* round-tripping. All procedures and globals are placed > correctly, and there are only one or two compiler niggles left. But a > very small number of jump table entries are getting transposed, > resulting in a few red speckles in vbindiff. Does it work as the Process Manager? It?s hard to believe there aren?t pragmas or command line options for setting jump table order. From elliotnunn at fastmail.com Mon Dec 31 01:18:21 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 01:18:21 +0800 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Message-ID: <8BA0D976-0AD6-4853-B97D-CEDA636CBCF0@fastmail.com> Not that I can find. I was happy to learn that segment renaming (-sg/-sn) and segment attribute-setting (-ra) arguments to Link can be used to enforce a segment order. (Officially there is no way to specify an order.) Normally jump table entries are ordered by segment, and then by the offset of the module within the segment. But when a module has multiple entry points, I have difficulty predicting the order. (Thankfully it is stable across repeated linker invocations.) > On 31 Dec 2018, at 1:12 am, daniel B-J wrote: > >> *Almost* round-tripping. All procedures and globals are placed >> correctly, and there are only one or two compiler niggles left. But a >> very small number of jump table entries are getting transposed, >> resulting in a few red speckles in vbindiff. > > Does it work as the Process Manager? > > It?s hard to believe there aren?t pragmas or command line options for setting jump table order. > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Mon Dec 31 01:19:15 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 01:19:15 +0800 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> Message-ID: <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> I'm too scared to try. :P Still need to tweak my Rez post-processor script a bit. > On 31 Dec 2018, at 1:12 am, daniel B-J wrote: > > Does it work as the Process Manager? From danielbj314 at verizon.net Mon Dec 31 01:26:47 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Sun, 30 Dec 2018 12:26:47 -0500 Subject: [CDG5] Process Manager In-Reply-To: <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> Message-ID: <31ADE966-C18F-4389-B1F5-EE96FF0A50F0@verizon.net> > I'm too scared to try. :P > > Still need to tweak my Rez post-processor script a bit. Maybe when it isn't like midnight for you (your dedication is impressive), you should write a tool for working with System Files. Maybe take a known good System File, overwrite some parts of it (isn't that what an enabler does? Could do something clever there), and stick the result in a image for miniVMac to try. From elliotnunn at fastmail.com Mon Dec 31 11:03:12 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 11:03:12 +0800 Subject: [CDG5] Process Manager In-Reply-To: <31ADE966-C18F-4389-B1F5-EE96FF0A50F0@verizon.net> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> <31ADE966-C18F-4389-B1F5-EE96FF0A50F0@verizon.net> Message-ID: <1BB7E271-B16C-4C81-86DB-F3E150BB41AF@fastmail.com> Got it! Things that didn't change the layout of jump table entries: - order of EntryPoint records in .o files - order of references made to EntryPoints - linker version - swearing In the end I just trawled through the manpage for Link and tried everything (thanks Jason). The -map option (I was using the -l option for my link maps) apparently has a subtle side effect, and fixed my jump table. Unfortunately there is very little segmented code for which we have any source, so this is not exactly a transferrable skill. Getting a perfect binary output isn't *entirely* self-indulgence on my part. It allows us to start and patches or mods from a known-good version of our source. Daniel, there is actually code here to build a ProcessMgrINIT. I'll get that working and try it out. > On 31 Dec 2018, at 1:26 am, Daniel B-J wrote: > > Maybe when it isn't like midnight for you (your dedication is impressive), you should write a tool for working with System Files. Maybe take a known good System File, overwrite some parts of it (isn't that what an enabler does? Could do something clever there), and stick the result in a image for miniVMac to try. From elliotnunn at fastmail.com Mon Dec 31 11:16:03 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 11:16:03 +0800 Subject: [CDG5] Process Manager In-Reply-To: <1BB7E271-B16C-4C81-86DB-F3E150BB41AF@fastmail.com> References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> <31ADE966-C18F-4389-B1F5-EE96FF0A50F0@verizon.net> <1BB7E271-B16C-4C81-86DB-F3E150BB41AF@fastmail.com> Message-ID: Sorry about the spam... It could be useful to write something to produce file x segment cross-references like this one I did by hand. . .M....... Error.a . .M....... OSDispatch.a . IM...K... ProcessMgrMisc.a . .....K... Switch.a . .M....... ZoomRect.a . IM...K.E. Startup.c . .....K... AppleEventExtensions.c G ......... Data.c . .....K..D Debugger.c . .M....... DeskMgrPatches.c G .......E. Eppc.c . IM......D Error.c G .M....... EventMgrPatches.c G IM....... FileSystem.c . .....K... HList.c G .....K... LayerMgrPatches.c . .M....... Memory.c G IMZ...... MemoryMgrPatches.c . .....K... MemoryMgr24Patches.c . ...3..... MemoryMgr32Patches.c G IM...K... MenuMgrPatches.c G .....K... OSDispatch.c . .M....... PackageMgrPatches.c G IM....... Patches.c G .....K... Processes.c G IM....... Puppet.c G .M....... Queue.c . .M....... ResourceMgrPatches.c . .....K... Schedule.c . .M...K... ScrapCoercion.c . IM...K... SegmentLoaderPatches.c . .M....... Sleep.c . IM...K... Switch.c . .....K... Utilities.c . .M....... WindowMgrPatches.c . .M....... Interface.o . ......... Runtime.o | ||||||||| . ......... Debugger . ........ eppc_segment . ....... %A5Init . ...... kernel_segment . ..... zone24_tools . .... zone32_tools . ... zone_tools . .. Main . . INIT . global > On 31 Dec 2018, at 11:03 am, Elliot Nunn wrote: > > Got it! > > Things that didn't change the layout of jump table entries: > > - order of EntryPoint records in .o files > - order of references made to EntryPoints > - linker version > - swearing > > In the end I just trawled through the manpage for Link and tried > everything (thanks Jason). The -map option (I was using the -l option > for my link maps) apparently has a subtle side effect, and fixed my jump > table. Unfortunately there is very little segmented code for which we > have any source, so this is not exactly a transferrable skill. > > Getting a perfect binary output isn't *entirely* self-indulgence on my > part. It allows us to start and patches or mods from a known-good > version of our source. > > Daniel, there is actually code here to build a ProcessMgrINIT. I'll get > that working and try it out. > >> On 31 Dec 2018, at 1:26 am, Daniel B-J wrote: >> >> Maybe when it isn't like midnight for you (your dedication is impressive), you should write a tool for working with System Files. Maybe take a known good System File, overwrite some parts of it (isn't that what an enabler does? Could do something clever there), and stick the result in a image for miniVMac to try. > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 From elliotnunn at fastmail.com Mon Dec 31 13:51:31 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 31 Dec 2018 13:51:31 +0800 Subject: [CDG5] Process Manager In-Reply-To: References: <36580478-8B6B-4D13-BE26-B5B76AEA6C0E@fastmail.com> <15EB032C-E968-4903-BF5A-9177F1907A40@fastmail.com> <4D40EE78-3084-411A-95C7-BC84BB8AF1C9@fastmail.com> <31ADE966-C18F-4389-B1F5-EE96FF0A50F0@verizon.net> <1BB7E271-B16C-4C81-86DB-F3E150BB41AF@fastmail.com> Message-ID: <770CA0E9-79A2-4A77-AB2D-63E3F25A99B2@fastmail.com> To build the ProcessMgrINIT, check out the 'system' branch of the CubeE repo. With BuildCubeE: :Build:Make System -src Src: Src:BuildResults:System:ProcessMgrINIT I tested the INIT on System 7.5 (in empw under Mini vMac). I know it works because if I slip a ShutDwnPower() into c_exittoshell(), the emulator shuts down whenever an app quits. But under SheepShaver the machine crashes after the splash screen, which is quite reasonable considering that I ripped out the embryonic 'PsychicTV' code in order to match System 7.1. The 68k 'scod' Process Manager is present at least as late as Mac OS 8.6, indicating that it was not a high priority for the 8.5-vintage PowerPC rewrites. Time for me to focus on other things for a bit, but (cough Daniel) I'd love to hear about any testing or tinkering done. All right -- I will quiet down about the Process Manager now. Happy new year everyone! > On 31 Dec 2018, at 11:16 am, Elliot Nunn wrote: > > Sorry about the spam... > > It could be useful to write something to produce file x segment > cross-references like this one I did by hand. > > . .M....... Error.a > . .M....... OSDispatch.a > . IM...K... ProcessMgrMisc.a > . .....K... Switch.a > . .M....... ZoomRect.a > . IM...K.E. Startup.c > . .....K... AppleEventExtensions.c > G ......... Data.c > . .....K..D Debugger.c > . .M....... DeskMgrPatches.c > G .......E. Eppc.c > . IM......D Error.c > G .M....... EventMgrPatches.c > G IM....... FileSystem.c > . .....K... HList.c > G .....K... LayerMgrPatches.c > . .M....... Memory.c > G IMZ...... MemoryMgrPatches.c > . .....K... MemoryMgr24Patches.c > . ...3..... MemoryMgr32Patches.c > G IM...K... MenuMgrPatches.c > G .....K... OSDispatch.c > . .M....... PackageMgrPatches.c > G IM....... Patches.c > G .....K... Processes.c > G IM....... Puppet.c > G .M....... Queue.c > . .M....... ResourceMgrPatches.c > . .....K... Schedule.c > . .M...K... ScrapCoercion.c > . IM...K... SegmentLoaderPatches.c > . .M....... Sleep.c > . IM...K... Switch.c > . .....K... Utilities.c > . .M....... WindowMgrPatches.c > . .M....... Interface.o > . ......... Runtime.o > | ||||||||| > . ......... Debugger > . ........ eppc_segment > . ....... %A5Init > . ...... kernel_segment > . ..... zone24_tools > . .... zone32_tools > . ... zone_tools > . .. Main > . . INIT > . global > >> On 31 Dec 2018, at 11:03 am, Elliot Nunn wrote: >> >> Got it! >> >> Things that didn't change the layout of jump table entries: >> >> - order of EntryPoint records in .o files >> - order of references made to EntryPoints >> - linker version >> - swearing >> >> In the end I just trawled through the manpage for Link and tried >> everything (thanks Jason). The -map option (I was using the -l option >> for my link maps) apparently has a subtle side effect, and fixed my jump >> table. Unfortunately there is very little segmented code for which we >> have any source, so this is not exactly a transferrable skill. >> >> Getting a perfect binary output isn't *entirely* self-indulgence on my >> part. It allows us to start and patches or mods from a known-good >> version of our source. >> >> Daniel, there is actually code here to build a ProcessMgrINIT. I'll get >> that working and try it out. >> >>> On 31 Dec 2018, at 1:26 am, Daniel B-J wrote: >>> >>> Maybe when it isn't like midnight for you (your dedication is impressive), you should write a tool for working with System Files. Maybe take a known good System File, overwrite some parts of it (isn't that what an enabler does? Could do something clever there), and stick the result in a image for miniVMac to try. >> >> _______________________________________________ >> cdg5 mailing list >> cdg5 at ucc.asn.au >> https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5