From elliotnunn at fastmail.com Mon Oct 8 00:07:06 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 8 Oct 2018 00:07:06 +0800 Subject: [CDG5] Pure Python HFS implementation Message-ID: Fellow hackers, Tonight I got my pure-Python HFS implementation to output a volume that can be mounted by Mac OS 9 and X (10?). Not much more work remains before it is solid enough to use in a build system. A slightly further-out goal is to expose the metadata underlying the Finder's spatial metaphor: XY coordinates etc. https://github.com/elliotnunn/machfs/ From maximumspatium at googlemail.com Thu Oct 11 23:25:21 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Thu, 11 Oct 2018 17:25:21 +0200 Subject: [CDG5] Pure Python HFS implementation In-Reply-To: References: Message-ID: Hi Elliot, thank you for keeping us updated and for your impressive work on HFS in Python. Could you elaborate the reason why do you need to maintain your own HFS implementation? Is there nothing we can reuse/improve? Am So., 7. Okt. 2018 um 18:07 Uhr schrieb Elliot Nunn < elliotnunn at fastmail.com>: > Fellow hackers, > > Tonight I got my pure-Python HFS implementation to output a volume that > can be mounted by Mac OS 9 and X (10?). Not much more work remains > before it is solid enough to use in a build system. > > A slightly further-out goal is to expose the metadata underlying the > Finder's spatial metaphor: XY coordinates etc. > > https://github.com/elliotnunn/machfs/ > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5 > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181011/c2d984e3/attachment.htm From elliotnunn at fastmail.com Fri Oct 12 22:10:32 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Fri, 12 Oct 2018 22:10:32 +0800 Subject: [CDG5] Pure Python HFS implementation In-Reply-To: References: Message-ID: Good question! First, I think it is pretty clear that we need software that can run on modern machines, and read and write disk images in a format native to the Classic Mac OS. This includes access to Mac-specific metadata, particularly resource forks and type/creator codes. Until now I have used Robert Leslie's hfsutils package in this role. It is a C implementation of HFS with several Tcl wrapper programs. My experience with empw (https://github.com/elliotnunn/empw) showed that the Tcl wrapper programs are very tedious to use in automated fashion. My very slow solution requires (deprecated features of) a recent macOS version, a non-stock version of rsync, deprecated Xcode command line tools, etc, etc. So I wrote some Python to expose the low-level "libhfs" part of hfsutils. This works well enough but really wants an object-oriented wrapper. Going through the libhfs code I realised that, although it is pretty well structured for C code, it makes some assumptions about its host that are no longer correct or helpful. In particular, it assumes that a filesystem is too big to fit in the process address space and so must be kept on disk and manipulated incrementally using IO calls, with all the inconvenience and complexity that that entails. As a result the code is quite long (~4700 lines excluding headers) and, to me, nearly inscrutable. So a wrote a Python implementation that just serialises and deserialises disk images in memory, leaving all the hard work of maintaining low-level data structures to Python's builtins. Currently it is 490 lines long. `import machfs` away! > On 11 Oct 2018, at 11:25 pm, Max Poliakovski wrote: > > Hi Elliot, > > thank you for keeping us updated and for your impressive work on HFS in Python. > > Could you elaborate the reason why do you need to maintain your own HFS implementation? Is there nothing we can reuse/improve? From elliotnunn at fastmail.com Wed Oct 24 18:20:37 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Wed, 24 Oct 2018 18:20:37 +0800 Subject: [CDG5] Python resource fork library Message-ID: <10067787-3390-4E7A-BFD0-E0F5E46DF92A@fastmail.com> Hi all, Here is another bit of yak-shaving. A Python library for resource fork making/breaking. SimpleRez and SimpleDeRez are actually faster than the originals. https://pypi.org/project/macresources/ https://github.com/elliotnunn/macresources From danielbj314 at verizon.net Thu Oct 25 07:32:45 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Wed, 24 Oct 2018 19:32:45 -0400 Subject: [CDG5] Python resource fork library In-Reply-To: <10067787-3390-4E7A-BFD0-E0F5E46DF92A@fastmail.com> References: <10067787-3390-4E7A-BFD0-E0F5E46DF92A@fastmail.com> Message-ID: <17593E8D-4C57-4342-8266-BC0AF45216EA@verizon.net> > A Python library for resource fork > making/breaking. Cool. Would I be correct in assuming it's mainly for use with your new CubeE build system? From danielbj314 at verizon.net Thu Oct 25 07:45:31 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Wed, 24 Oct 2018 19:45:31 -0400 Subject: [CDG5] Building OF Client Interface Programs Message-ID: <6C6DE36C-3BCC-4BD0-94F4-262CB0CA68FA@verizon.net> Hello Everyone, I am having a bit of a problem with building Client Interface Programs. XCOFF files (the only overlap between what PPCLink can make and what Open Firmware can parse) are not relocated by Open Firmware. I have to specify the address ahead of time. Even worse, Open Firmware insists on placing them at the physical address they specify, in addition to the virtual address specified in XCOFF sections. Additionally, I want the program I am making to not overlap with the Trampoline, so address ranges 0x00100000-0x0019967 and 0x00200000-0x0021025F aren't usable either. There are also some issues with the parcels image sometimes taking the memory I want. Any ideas on what I could do to make my code usable on as many NewWorld machines as possible? I could rewrite the c as fcode, but that would be hard and annoying. Thanks, Daniel From maximumspatium at googlemail.com Thu Oct 25 08:03:04 2018 From: maximumspatium at googlemail.com (Max Poliakovski) Date: Thu, 25 Oct 2018 02:03:04 +0200 Subject: [CDG5] Building OF Client Interface Programs In-Reply-To: <6C6DE36C-3BCC-4BD0-94F4-262CB0CA68FA@verizon.net> References: <6C6DE36C-3BCC-4BD0-94F4-262CB0CA68FA@verizon.net> Message-ID: Am Do., 25. Okt. 2018 um 01:45 Uhr schrieb Daniel B-J < danielbj314 at verizon.net>: > Hello Everyone, > > I am having a bit of a problem with building Client Interface Programs. > XCOFF files (the only overlap between what PPCLink can make and what Open > Firmware can parse) are not relocated by Open Firmware. I have to specify > the address ahead of time. Even worse, Open Firmware insists on placing > them at the physical address they specify, in addition to the virtual > address specified in XCOFF sections. > > Additionally, I want the program I am making to not overlap with the > Trampoline, so address ranges 0x00100000-0x0019967 and > 0x00200000-0x0021025F aren't usable either. There are also some issues with > the parcels image sometimes taking the memory I want. > > Any ideas on what I could do to make my code usable on as many NewWorld > machines as possible? I could rewrite the c as fcode, but that would be > hard and annoying. > It's interesting that XCOFF in OF isn't relocatable. I remember reading somewhere that earlier Apple OF implementation used to include very primitive loader packages. I was told that this has been fixed in later versions. Is that true for any specific OF version or for ALL NW versions? This is probably the reason why the Trampoline was supplied as ELF container. I wonder how Apple's engineers performed XCOFF to ELF conversion. MPW, used to compile the Trampoline, cannot produce ELF, at least not with the publicly available release. My bet is that a custom conversion tool similar to macho-to-xcoff in BootX has been used for that purpose. I assume that non-relocatable XCOFF doesn't present a problem in BootX... -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181025/f5d64d10/attachment.htm From danielbj314 at verizon.net Thu Oct 25 08:47:12 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Wed, 24 Oct 2018 20:47:12 -0400 Subject: [CDG5] Building OF Client Interface Programs In-Reply-To: References: <6C6DE36C-3BCC-4BD0-94F4-262CB0CA68FA@verizon.net> Message-ID: <41B48B0D-A1C0-4D90-85B1-C0E5182EDD96@verizon.net> > It's interesting that XCOFF in OF isn't relocatable. I remember reading somewhere that earlier Apple OF implementation used to include very primitive loader packages. I was told that this has been fixed in later versions. > > Is that true for any specific OF version or for ALL NW versions? It's definitely true for the Pismo I do most of my hacking on. Its xcoff-loader package has very simple code. I'm still not sure if the elf-loader is any different. The code is much more complex and I can't follow it well. At least the ELF doesn't actually specify a physical address (according to the manually-layed-out ELF header for the Trampoline in the original cdg5 repo). > I assume that non-relocatable XCOFF doesn't present a problem in BootX... The boot code doesn't do anything weird like modify the XCOFF, so they solved that problem. Probably by using 0x00100000 and 0x00200000. > My bet is that a custom conversion tool similar to macho-to-xcoff in BootX has been used for that purpose. Probably. I don't really know the format for either XCOFF or ELF. From danielbj314 at verizon.net Fri Oct 26 08:51:00 2018 From: danielbj314 at verizon.net (Daniel B-J) Date: Thu, 25 Oct 2018 20:51:00 -0400 Subject: [CDG5] Building OF Client Interface Programs In-Reply-To: <41B48B0D-A1C0-4D90-85B1-C0E5182EDD96@verizon.net> References: <6C6DE36C-3BCC-4BD0-94F4-262CB0CA68FA@verizon.net> <41B48B0D-A1C0-4D90-85B1-C0E5182EDD96@verizon.net> Message-ID: <8FFA1E92-AA04-4FF6-813E-31D1FD60BA99@verizon.net> I did a few tests with fcode, and I think it is definitely the way to go for what I want. It has no particular memory mapping requirements, is supported on all Mac OF implementations, and is trivially callable from a tbxi bootscript with half a line of code: load-base {fcode-offset} + 1 byte-load It should even work on Elliot's G4, which is kind of famous for blowing up with slight boot-script changes. It can't complain about the syntax with everything already compiled into bytecode. The one downside is that fcode is in forth and my gui is written in c. The conversion will not be fun. But that's the price to pay for pretty-much guaranteed compatibility with all NewWorld systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/cdg5/attachments/20181025/a1e47afb/attachment.htm From elliotnunn at fastmail.com Fri Oct 26 15:04:43 2018 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Fri, 26 Oct 2018 15:04:43 +0800 Subject: [CDG5] Python resource fork library In-Reply-To: <17593E8D-4C57-4342-8266-BC0AF45216EA@verizon.net> References: <10067787-3390-4E7A-BFD0-E0F5E46DF92A@fastmail.com> <17593E8D-4C57-4342-8266-BC0AF45216EA@verizon.net> Message-ID: <44CF50F0-B8EB-43C1-A6F4-DE9CD2538227@fastmail.com> Yes... but I think it was a missing piece in the broader MacOS reversal/development toolkit. That?s nearly enough mucking around with CubeE. Back to the NanoKernel soon, to see what can be done with a G5! > On 25 Oct 2018, at 7:32 am, Daniel B-J wrote: > > >> A Python library for resource fork >> making/breaking. > Cool. > > Would I be correct in assuming it's mainly for use with your new CubeE build system? > _______________________________________________ > cdg5 mailing list > cdg5 at ucc.asn.au > https://lists.ucc.gu.uwa.edu.au/mailman/listinfo/cdg5