Hello there. I thought I'd put together an easy-to-follow guide for preserving floppy disk media, as there seems to be growing interest in the subject. It feels not too long ago that information about this arcane, long-lost art was exceedingly scarce -- back when Old Krug was learning the ropes, nuggets of knowledge had to be excavated from ancient write-ups, whose authors' beards could be smelled through the screen. Add in a forum post here, a Discord message there, and you get the picture. While I'm grateful for everything I learned from Those Who Came Before Me, what I managed to find was only enough to get me started. Everything else I've had to figure out by myself, basically through endless trial and error. I like to think this journey has given me a somewhat unique perspective on things, so some of my preferred techniques, workflows, and opinions may strike you as hot takes. Rest assured, there's always a method to my madness, and I typically have numbers to back up my claims. With over 5000 disks under my belt, I hope my experiences can not only help you get going, but also put you on a track that leads to the best possible results, avoiding countless traps along the way. The focus here will be on Japanese computers from the 80s and 90s, as those are what I and people around me are most interested in, but the basic principles apply to other systems as well. Just be mindful of any special steps that may be needed to read certain types of disks (e.g. Commodore 64).
The guide is divided into three parts. We start out by getting familiar with the equipment needed for our tasks. There's a lot of ground to cover here, so in-depth discussions will center around things that may not be immediately obvious (or easily googleable). For step-by-step installation guides, spec sheets of specific hardware models, and similar things, there will be a list of links provided at the end. Next, we move on to preparing the disks: inspecting them for mold, scratches and other oddities, and cleaning the data "donuts" to the best of our ability. I consider this section the most important one, as it contains many original techniques and ideas I've come up with over the years (at the cost of my sanity, and heaps of trashed disks, some of which are too rare to think about💦). These should make your success rate increase dramatically with disks that are in less-than-ideal condition -- which is to say, a significant portion of the ones you'll be encountering in this day and age. Last, we go through the process of dumping some disks, hopefully in enough depth to cover all potential pitfalls. The unfortunate reality of dealing with 30-year old magnetic media is, a single wrong command input, or careless swipe with a cotton swab, can be enough to cause irreversible loss of data. So prepare yourself mentally, and buckle up for a wild ride.😎
NB: The information presented here is in a constant state of flux (pun very much intended), with new sections being added as time allows, and old ones rewritten as I gain more experience. Please try to refrain from quoting the guide like scripture, or trashing it because a technical detail is inaccurate. Let's have a conversation and make it better over time.✌️
As for preparing our drives, 3.5" ones should work pretty much out of the box. (The only hitch comes when you want a 3.5" J-PC drive to spin at 360rpm, which requires cutting/detaching Pin 1 on the ribbon cable.) With 5.25" drives, things get a bit trickier, as they often require changing jumpers to work for our use-case. I'll put up links to technical details of certain popular models, as well as pics showing how the jumpers are set on my units, at the end of this section. Whichever model you choose to get, make sure it has the jumper pins populated, unless you're technically savvy enough to get around such problems. I ran into this one with the TEAC FD-55GFR's in my X68000 Pro (which also had an electronic disk loading mechanism that had to be unscrewed for the lever to turn manually), whereas some other revisions of this model apparently don't have these issues, and come recommended by many users. My GFR also seems to handle bad condition disks really well, which, while unscientific due to low sample size, is worth mentioning. As a general rule, I think it's best to look for drives from well-known manufacturers like NEC and TEAC, as ones from lesser-known brands may be lacking in configurability and documentation. Without a scan of the user manual available, you can run into big problems with a mystery drive down the line.
Next, you're going to need a specialized controller device capable of reading magnetic flux transitions (think of these as a layer below the data) to interface with your drives, and produce archival-grade preservation copies of your disks, including any copy protection schemes that may be present. Nowadays the market for such devices is brimming with options, but frankly I think this is an easy one: you should grab a Greaseweazle, which is not only cheap and open source, but also relatively easy to set up and use. Any model will do, they can be found for around €20-30, and more technical folks can assemble an F1 for just a couple euros. As for "competing" products, KryoFlux has been around for a long time, but the tech is aging, the price is high, and the licensing scheme is apparently very contentious. (TBQH, if there's a single thing it does better than GW, I've not been made aware of it.) Pauline, in contrast, represents to me the pinnacle of dumping tech, but lacks the ability to write disks, reportedly costs
The most important first step is to inspect all our disks meticulously for anything that is "off", and clean the surfaces of the data "donuts" to the best of our ability. To accomplish this, we're going to need quite a bunch of stuff. First, find a 3D printer (I used the one at my local library) to make cleaning brackets for all form-factors you require. Below you see mine: one for 3.5" disks, and one for 5.25". Be smarter than I was, and print the loose "knob" in some gaudy color, to avoid losing it among your other thingamajigs... every. single. honking. time.
Two quick caveats: When I say scrub (throughout this guide), I don't actually mean using force. Using force here will make the sensor dislodge, so be very gentle (if firm). Then, when scrubbing the top head, press it from above with a finger to avoid lifting it up by the force of the scrubbing, which leads to misalignment, because you're putting more strain on the springs than intended. Realigning the heads is a pretty advanced procedure, so you'll want to avoid that if possible. That said, at least with 5.25" drives, it's not as bad as some people claim, so don't worry too much about it. I go through the steps here, in an addendum to the third chapter of the guide.
With this magic potion (or soapy water, as the next best option) secured, I think we've covered all our basic equipment needs.🕺 For more in-depth discussions on select topics covered in this chapter, check out the addendum below. Also, feel free to suggest more topics that you feel should be explored deeper at this point in the guide.
Chapter 1: Addendum
The state of floppy disk media
Before we move on to inspecting our disks, I should set the tone for what lies ahead of us. An aspiring preservationist in the 2020s will be encountering lots of disks that are in less than ideal condition. Even when buying software sold as tested, getting "lemons" is a very normal thing to happen. This is for a number of reasons: Testing (e.g. kidou kakunin / 起動確認 on YAJ) typically consists of inserting the first one or two disks, and playing the intro or first stage for five minutes, leaving whatever parts of the disk(s) that didn't get accessed, as well as all subsequent disks, untested. Furthermore, testing a dirty disk will make the dirt spin at 360rpm, which should strike you as terrifying -- even when the tester gets a read out of it, what comes out of the drive may be a scratched up mess. (Here's an example: YAJ pics vs. on arrival. Notice how those scratches seem to have appeared after the seller took the disk pic, suggesting they formed during a single round of testing.) Then there are shops and sellers who say they visually inspect their disks for mold and scratches. These practices, while generally great and worth paying a premium for, come with a healthy dose of subjective evaluations and outright negligence. It's basically a rite of passage to order a game with no marked defects on Suruga-ya (my favorite store), and get one with moldy disks (which should've been marked 動作不良品(カビ) under their own classification system). It happens to the best of us (regularly). And that being the reality with "tested" items, you can imagine how much worse things get with untested ones, which comprise the bulk of offerings on auction / flea market sites. (Spoiler alert: the situation is getting really bad. If you stick to buying pristine-looking / newer stuff, you'll get a better ratio than mine, but at the time of writing this in 2024, roughly every second game I grab from YAJ / Mercari has disks that need cleaning, with more and more of them requiring major operations to salvage.) Don't let any of this discourage you, though. I mainly want to hammer in from the start perhaps the most important lesson you can learn from this guide: it's absolutely crucial to inspect your disks thoroughly, and clean them up as needed, before putting them anywhere near a drive.
Honestly, I can't overemphasize how important this point is, and it's something I don't see talked about much elsewhere, so bear with me a bit longer. Disks you buy from auctions have typically been sitting in someone's mom's attic (jp: 実家) or similar type of storage for years -- quite possibly decades. They will have gathered dust, dirt and mold (among myriad other defects). However, the fact that they've sat there untouched is arguably the main reason why their data is still likely to be intact. This is because, based on what I've seen, the aforementioned things by themselves don't seem to be enough to ruin the disks (excluding certain extreme situations, environments, and levels of neglect). The main killer of data is something quite different: the user, who puts a dirty disk in a drive "to see if it works", or is too impatient with their cleaning methods, producing scratches where the mold used to be. To put it in other words, in many if not most cases, the person who gets the first dibs to kill these disks is you. And you will be doing your fair share of killing, let me assure you of that. This guide can only help you minimize your losses.
2D vs 2HD native drives for 360KB 2D disks (e.g. PC-88, X1, FM-7)
I know this is jumping ahead to technical stuff we haven't covered yet, but anticipating resistance to my claim that drive type makes a real difference when dumping 360KB 2D disks, I want to present two pictures showing how big of a thing I'm actually talking about. In the top one, I've dumped a disk using FD-1155D (a 2HD drive) in DD mode, with double-stepping: gw read --dd=L --tracks=c=0-41:step=2 dump.scp A standard opinion (even among experienced users) is that this is about as good as it gets, as DD mode adjusts the "write bias current", elevating 2HD drives on par with 2D native ones. With that in mind, let's take a look at the bottom one, where I've dumped the same disk on a TEAC FD-55BR. I'm far from having any kind of engineering competence, but it seems to me like the wider heads of 2D native drives are making an astonishing difference, in terms of being able to read around scratches. I've heard arguments saying these comparisons can also go in favor of 2HD drives, due to their more advanced sensors getting better reads than their 2D counterparts, but I honestly don't see this playing out in practice. Meanwhile, my 2D native drive is producing jaw-dropping improvements, like you just witnessed there. Just wanted to make sure my findings are out there for everyone to see, and judge for themselves.
Chapter 1: Recommended reading
Greaseweazle models
Greaseweazle wiki
Cleaning brackets (and other useful 3D printables)
Seller of cleaning diskettes
NEC FD-1155D technical details
TEAC FD-55GFR spec sheet
TEAC FD-55GFR jumpers
My jumper settings:
NEC FD-1155D
TEAC FD-55GFR
TEAC FD-55BR
My basic method of cleaning through the hole, window, whatever you call it, is to first apply electrolyzed water using a cotton swab. Dip the swab long enough for it to become thoroughly moist. Then tap the problem spot on the disk ever so lightly, just enough for a puddle of water to encompass it. Don't leave it too dry, as the increased friction will lead to more oopsies, i.e. scratches developing when scrubbing. Equally important is to not get the spot on the disk too wet, as you want to avoid fluid getting under the sleeve, where it becomes painfully difficult to remove. (You basically have to keep spinning the disk by hand, and drying the area visible through the window until the moisture runs out.) Recall from earlier that spinning a disk with a wet liner leads to Bad Things, so you want to avoid getting to this situation as much as possible. Now then, let the puddle sit there for a while. If the problem spot looks tough, there are only positives to waiting longer, so don't rush things here. Especially with disks that have only a few problem spots, letting them sit for a couple minutes per spot won't kill you. Even with "easy" spots (e.g. with light white mold), thirty seconds is a lot better than five. As you gain more experience, you'll get a feel for what trade-offs you're making (stochastically speaking, a.k.a. in the long run) when cutting corners in different situations, and can start to make educated decisions on when to go faster. But it's best to start out real slow. If time is money to you, this is just about the worst hobby you could've chosen. I'd honestly consider quitting while you're still ahead :^)
Here's a list of things you can expect to see on the disk's data surface:
NB: I'm going to be adding more pics showing more mold varieties under my loosely defined categories in the weeks and months to come. Don't take the current ones to mean those are the only types I've taken into account. They just happen to be what I found on my phone at the time of this writing :)
This is the type of mold you're glad to see, not only because it comes off easily, but also because it scares away less experienced buyers in the auctions, letting you grab expensive software for jaw-dropping discounts. If you save as much money as I have using this tip, consider making a donation to Old Krug's
When there's so much mold that it becomes difficult to clean through the window, don't spend hours trying to get it done. Instead, scroll down to the section where I describe the process of desleeving, i.e. taking the data donut out of the casing, and cleaning it from both sides in one go.
Easily recognizable from the hardened snowflake (or "frost flower", as we call window frost in Finland) patterns, this type of mold is still nothing to be afraid of. Just let it soak for a good bit longer than the previous kind. Removal may leave an ugly stain behind, but this shouldn't affect reading, as long as you didn't produce scratches. When light scrubbing isn't enough to get the mold to come off, let it soak for longer; and when there's too much "frost" to clean through the window, desleeve without hesitation.
Oh hey, it's your lucky day: someone left a pile of coke on your disk (for a decade). This type of mold, which has grown/hardened/condensed to the point of acquiring 3-dimensionality (a.k.a. height), comes with a high risk of bad things happening as it is removed, so be sure to take extra long when soaking it, to maximize the chances of it coming off without scratching, or taking away the top layer with it.
The prognosis will vary based on how well the removal went, but even when you get a stain as nasty as this one (from the mold in the picture above), it's possible the disk will read fine, as was the case here. (Perhaps needless to say, the odds get a lot worse on 3.5", as I'll explain here.)
Light spots of brown mold, like their white brethren, come off easily. Things only get complicated when there's more of it than can be easily cleaned through the window. We'll talk about that scenario next.
When there's enough brown stuff to consider it less like spots, and more like a coat, it can sometimes be hard to decide on the right approach. As a general rule, for mold coats of any kind, the only good approach is to desleeve the disk, which is an involved process -- one that I consider safe when done properly, but there are definitely risks involved. As such, there's an argument to be made for trying to read specific types of iffy-looking disks as is. The thing you need to ask yourself is, are you really witnessing a mold coat, or something that looks similar but is actually different, like a brownish tint, or a pattern across the disk? I'm not sure what causes all these phenomena, but there seem to be disks that at first glance look moldy, but may not in fact benefit much from desleeving. When in doubt, try to clean a bit of the disk through the window, and see if it makes a clear difference. If it does, congrats, it's mold, and you should skip to desleeving. If cleaning doesn't change how the disk looks, you may be looking at one that you should consider reading as is. Reading a disk with weird brown patterns is a bit unnerving, of course, and bad condition disks are known to make all kinds of offputting noises during the reading process (some of which are fine and normal, while others mean you need to hit the emergency brake). If you're just starting out, I recommend putting disks like these in a problem pile, to be tackled after developing a good feel for this stuff. You don't want to destroy a disk that could've been salvaged using other methods (or having better situational awareness), by being impatient and yoloing it.
Luckily, when a brown mold coat gets nasty enough, there's no mistaking it for anything else. It's quite likely the mold has eaten into the top layer(s) at this point, and removing it will leave your disk in a very precarious state (at best). Desleeved cleaning, followed by a prayer, is the only option here. Be sure to monitor the dump in progress, and stop the read process immediately when you see an issue in the pie charts. (Often, it'll be too late at that point, but that's something a preservationist has to accept, and learn to live with.)
PS. For disks like this, which have lost their luster after cleaning, experienced users have reported success applying a light coat of cyclomethicone as a lubricant, but that process comes with its own set of challenges, and I lack any first hand experience at the time of this writing, so I hesitate to say much about the subject. I'll try to cover the topic in a dedicated segment down the line.
Scratches
Did you think mold is bad? That was actually the fun part. From here onward, we'll be talking about the real killers. You see, generally speaking, mold is something that grows on top of the disk surface, letting you wipe it off (given the right technique), whereas scratches cleave into it, physically killing the data underneath. Luckily, in most cases they're not a complete death sentence. Light ones can often be read without issues; and the less densely packed the data is (e.g. 360KB 5.25" 2D vs. 1.44MB 3.5" 2HD), the larger the chances of there being slivers of readable data around the damaged area (as data tracks tend to be wider than the typical speck of crud that cleaved your disk). Even when you get no luck with your best performing drive, through experimenting with realigning drive heads by hand, I've discovered there's still hope left: As the last nuclear option, it's possible to deliberately misalign a drive a bit, or wiggle the carriage around slightly during read attempts, to get full track width reads from the problem spots (the technical term for this is "microstepping"). The "gains" from "manual microstepping" are in fact quite astonishing, so be sure to check out my dedicated segment about it.
As you can tell from the shape, scratches like these are created while the disk is spinning. When there's crud on the disk, and you make it spin at 360rpm, two very bad things will happen: 1) the crud will rub against the liner (or, alternatively, a liner with crud lodged into it will rub against the disk); and 2) the read heads will come into contact with the crud, with some of it eventually getting stuck between the head and the surface, forming a contraption that starts to "plough" a "furrow" on your disk (if you city slickers follow my epic rural analogy. 🐓)
There's no way to fix a scratch, so our options are fairly limited here. First, we want to clean both the scratched area and the drive heads as well as we can (out of dirt and debris from the top layers). This is best done by desleeving the donut, as loose debris is bound to spread across the disk under the sleeve, ready to cause more issues (often in unexpected places). For the same reason, it's important to replant the scratched donut into a clean sleeve after cleaning. Then we just try to read the scratched tracks again, and hope it goes better this time. If you stopped the drive fast enough when the scratch started to form (=while the scratch is still black to gray color), the prognosis is quite good. As for deeper (=white) scratches, often times the only move is to try reading around it (via microstepping), and hope there are slivers of data from the affected track left alive either before or after the scratch.
These types of scratches are typically formed during cleaning. Perhaps you were too impatient while soaking the mold, and the removal produced scratches? Maybe you scrubbed the disk using something too coarse, like a non-ultrafine microfiber or a cotton swab? Or perhaps the disk was left on a desk, and something cleaved the spot exposed through the window? In any case, proceed like we did with circular scratches in the previous segment. The scary thing about radial ones is, if they're "perfectly" radial, they can affect a track (or as is often the case, several) for the entire width, making it impossible to microstep around the damage. That said, in most cases the scratches are at least somewhat angular to the track data, leaving room for microstepping to save the day. Also, sometimes you see a giant scratch affecting pretty much the entire radius of the disk, but it happens to be in the empty area between the first and last sectors, where there's no data for it to destroy. (E.g. this disk, for my Dead of the Brain II, somehow dumped without issue.) A preservationist will take all the lucky breaks they can get.🕺
Last, I should mention dents. Sometimes they're fine, (many) other times they're not. There's not much you can do about them, and there's no downside to trying to read them, so just try and see how it goes. One interesting thing to note is, as you can see by playing around with a desleeved disk, dents don't develop easily. A corollary of this is, even when you see disks with bent sleeves for sale, it's more than likely the donuts inside are fine, and can be read without issues when transplanted to a clean sleeve. (This may or may not be how I saved $100 on GaoGao! 3rd :^))
Desleeving
Taking the donut out of the sleeve is, in my experience, by far the best way to handle disks with more than a couple spots that need to be cleaned. While unnerving to newcomers, the process is safe when done properly, and even completely reversible with 3.5" disks. With 5.25" disks, there's a bit of destructive cutting involved; but insofar as your goal is to preserve the data (as opposed to nondescript plastic), the benefits outweigh the costs by a huge margin.
As the process differs based on form-factor, we'll be going through the steps for 3.5" and 5.25" disks separately below. The main idea is the same: we take the donut out of the sleeve, soak it in water for the crud to loosen its grip, wipe everything off gently with ultrafine microfiber, remove residue (water stains, dust specks, finger grease etc.), and replant to a clean sleeve. Once you get the hang of it, desleeving starts to feel very natural -- and seeing the results, something you wish you had done from day one lol. Based on my experiences, I honestly guarantee it.
Desleeving a 3.5" disk
Opening this type of a sleeve is an involved process, which is easiest to show in a video, so I hope my YouTube tutorial is ready by the time this guide goes live. If not, here's my best attempt at describing in words how to go about it: Start by sliding the door toward the open position, while pinching the sleeve from both sides at the spot where the spring is housed using your other hand. When the door reaches the spot you've been pinching, force it upward (from the spot where the spring is housed), and it should dislodge a bit. Let the door move back toward the closed position, and it should dislodge even further, letting you pull it out as you slide it open one last time. Make sure you get the spring out at this point, and try not to lose it. Next, pry open the top (=door) side of the sleeve from both corners. This can usually be done using your nails, but some disks are very rigid, requiring a knife or similar to cut through the plastic bits holding the front and back sides together. When you need to resort to a knife, be extremely careful to avoid cleaving the donut when the plastic gives in. When both corners are done, pull the sides apart enough for the donut to have room to come out, and let it drop in your hand. Handle the disk with your fingers on the hub, and avoid touching the donut itself.
NB: As noted earlier in the guide, using IPA will (arguably) always degrade the disk (on a microscopic level, at the very least). I consider this a special situation where the benefit far outweighs the cost, as I haven't seen another liquid do as good a job at residue / water stain removal, which is an absolutely crucial step to get done right, in order to get a good read of a disk that has gone through this type of a major cleaning operation. In the same vein, I've never observed reads get worse from this type of quick IPA spray use. But considering you will likely hear people scoff at the mention of IPA, I thought it best to include this caveat, so that everyone understands what we're doing here.
You'll be glad to hear, this type of sleeve is very straightforward to open. The only downside is, the operation will be destructive, in that you won't get a perfect sleeve back when we're done. I should note, it's *technically* possible to bend the top flap open (and back at the end), but in practice this never works out great (e.g. the plastic breaks off in random spots, the flap cleaves the donut as you take it out, and/or won't settle nicely back in place). Therefore, the only good way to do this in my experience, is to cut along the top edge of the sleeve. Note that there's precious little room (around 3mm) between the top edge and the donut residing inside. Be sure to press the donut against the bottom edge from the center hole, to make the gap as big as possible. Then try to get the cut done in big clean cuts, to avoid leaving pointy edges that may cleave the donut. When done, the donut simply slides out into your palm.
NB: When flipping the donut around, it's easy to lose track of which side is which. Putting it in the wrong way is no big deal; when it happens, just stop the read with ctrl-c, eject the disk, and flip the donut over. No harm done :)
Chapter 2: Addendum
[Cleaning-related topics shall be discussed here in depth]
Now, as if that wasn't enough of a headache, 5.25" drives also require an external power source, as Greaseweazle won't provide the needed 12V. Luckily there are many options here. I prefer to use a UGREEN USB3 adapter, which I'm already using for a Plextor optical drive. The adapter gets its power from the wall, and seems to provide enough juice for multiple drives operating simultaneously, which is quite wonderful. So I just attach a Molex splitter, and voilà:
Last, there's a third option of getting a dedicated Molex power supply. Just make sure the specs match what is written on GW wiki before ordering one. I recall hearing they don't necessarily come with 12V either.
Ok then, are we all set? I think we're ready to plug in the USB, and head on to command prompt to see if things are working: gw info should show your device as connected. (If you're having trouble getting your GW F1 to appear, remember that some Micro-USB cables are power-only, as they're only meant for charging old cell phones and such. Try a few different cables, and you should find one that has data too. I swear this tripped me up for quite a while...) Next, let's make sure we have the latest firmware for our device, and the latest version of GW tools: gw update takes care of the former, and the latter can be downloaded from here. Now then, are your disks and drive all cleaned up? We're going to start dumping!
As you'll see by typing gw --help, Greaseweazle presents us with an avalanche of options. (Luckily we only need a handful of them.) The choice of file type is perhaps the most consequential one. You'll want to start out by dumping your data in a flux format, which results in relatively large files: tens of megabytes per floppy disk image. Later you can convert these flux images to smaller, more convenient formats to be used on emulators, for example. But you can't do this the other way around, so let's be sure to preserve the flux information first. There are basically two flux formats to choose from: .scp and .raw. The former is my preferred one for archival and "everyday use", as it only produces a single file per disk. Meanwhile, the latter helps us in many ways to get good reads from less-than-ideal condition disks, thanks to it being separated into per-track files. So we'll be making use of both formats.
To create the best possible preservation copy of a disk, we're going to want to read every single track our drive is physically capable of accessing. This means tweaking the settings to include the very last tracks, which may be considered "outside of the spec", yet still contain data (e.g. copy-protection stuff, or miscellaneous metadata), as shown in the picture below:
To dump a 2D disk with 40 tracks, we need to specify the correct number, as GW default settings are for 80-track disks (making 40-track drive heads bang against the "wall" after crossing 41). When using a 2D native drive, it's as simple as: gw read --tracks=c=0-41 mydump00.0.raw. However, when using a 2HD drive, we need to set two more things: 1) double-step mode, to make the drive skip every second track, and 2) DD mode, to adjust the write-bias current. These tweaks essentially make the drive mimic a 2D native one: gw read --tracks=c=0-41:step=2 --dd=L mydump00.0.raw (The 0-41 track range covers the "extra" tracks we talked about earlier.)
Last, for 2DD disks (e.g. MSX and early PC-98 titles, typically in 3.5" form-factor), we leave out the double-stepping, as this format wants DD mode, while being 80 tracks in length. So: gw read --dd=L mydump00.0.raw
With our first clean 3-revolution .raw dump secured, we've made sure the whole disk is readable in one go, and the coast is now clear to make a final, 5-revolution dump for archival purposes. (For further details on why I do things this way, check out this section.) Use the same command you did the first time around, but specify --revs=5 and the .scp format: e.g. gw read --revs=5 mydump.scp for 2HD disks.
Honestly, that's pretty much the gist of it. There's still a wide range of advanced topics left to cover, which I'll hopefully get to in the coming weeks and months. The first one that comes to mind is how to recognize and deal with various types of copy protection schemes that you'll no doubt soon start encountering. I'm going to start writing about it below, but leave you with some parting words already at this point:
While some sections of the guide still need to be fleshed out more, to include more examples, varieties, and all that good stuff, if you've made it this far, you should now have all the tools needed to get good dumps going. Be sure to check back for updates, and revisions to what's already been written, based on feedback I get going forward.
Peace & Love✌️
- Krug
Copy protection
As I'm sure everyone reading this is aware, back in the day, programmers came up with countless clever ways to stop people from copying those floppies. Typically, a specific section of the data was crafted so that it got garbled when copied to a new disk (using standard methods of the time). What this looks like in practice, i.e. when looking at the pie charts, is often deceptively similar to what you see in dirty or damaged areas (based on the criteria laid out in the previous section of this guide), potentially leading to all kinds of confusion, especially when trying to assess the condition of a disk's data surface. Luckily, copy protection areas come with several peculiarities, which let an astute observer tell them apart from read errors caused by dirt or damage.
The most important one of these is patternality: Protection schemes tend to produce patterns (or signatures, as I often call them) that you'll soon start to recognize, simply because they appear so often, and always look the same. (NB: The lack of a signature doesn't necessarily mean the software is unprotected.) Another important one is location: At least on J-PC games, protection patterns tend to reside either at the beginning (first 10 tracks or so), or at the very end (around track 76 on 2HD disks, and 39 on 2D ones -- or even further, in the "outside of spec" areas, namely tracks 77-81 (2HD) and 40-41 (2D), which aren't used for normal data on these systems.) This means that if you spot iffy sectors around the middle of the disk, you're almost certainly looking at unintentional read errors from dirt/damage. (NB: There's always an exception that proves the rule...) You'll come to see that this type of "if it's not this, then it must be that" thinking is very helpful in figuring this stuff out. Outside of the pie charts themselves, the number of sectors (especially good vs bad ones), given at the top left corner of the disk visualization view, is another factor that often gives the game away. However, the logic there takes a bit more room to explain, so I'll be talking about it in a dedicated section.
With these three concepts -- patterns, location, and number of sectors -- you should have everything needed to figure out when an iffy-looking section is intentional (protection), and when it is unintentional (dirt/damage).
To get you started with pattern recognition, I've compiled a small list of common copy protection signatures that you run into with J-PC games:
11 bad sectors on both sides
5 bad on Side 0 / 6 bad on Side 1
1 bad / 1 bad
0 bad / 4 bad
3 bad on Track 77.0
PS. I'd love it if someone more technical than me could figure out what the names of these protection types/schemes are, as well as any other details that pertains to them. With our powers combined, we could make a cool database out of this!
Chapter 3: Addendum
Advanced techniques for tackling problem spots
[Coming Soon(tm)]
3.5" vs 5.25": The Form-factor Debate
[Coming Soon(tm)]
Manual Microstepping (feat. The Wiggle Method)
When a disk has scratches that are deep enough to prevent you from getting a good read of the damaged areas, the only move is to try reading around them, hoping there are slivers of data from the affected track(s) left alive either before or after (i.e. over or under) the scratches. I wouldn't blame anyone for thinking this sounds like a long shot, but incredibly enough, based on my findings, it's actually very likely that you can salvage the situation by doing just that; namely, moving the heads in fraction-of-a-track-width increments, which is a process known as microstepping. My best educated guess for why this works so often is as follows: The speck of crud that got caught in the read head (or whatever), and produced those scratches, likely was smaller in diameter than the full width of a data track, which have leeway built in, so that slight imperfections in head alignment don't automatically result in bad reads. To add to that, the bigger the disk's form-factor, the wider the tracks, and the smaller the area (as a percentage of track width) that any particular size of crud will affect...
Following this logic, one comes to an unintuitive conclusion, which is that the most robust disks against damage are in fact some of the oldest: Indeed, I've found 360KB 2D 5.25" to be the gold standard in this regard, as it benefits not only from the bigger diameter, but also from the sparseness of data. Honestly, I'm not sure I've ever failed to salvage a disk in this form-factor -- which is a statement that sounds utterly preposterous, I know. But what am I supposed to do -- lie? 🤣
So, how does one go about "microstepping", then? The prevailing notion is that you need to equip your drive with a specialized motor, capable of moving in tiny steps. The details on what to buy from where, and how to set things up were never clear to me, so what I did was, I decided to skip this stuff altogether, and see if I could make those microsteps happen manually. My first idea was to loosen the screws, and deliberately misalign the heads a tiny bit, to get reads from different parts of the tracks. This seemed to work pretty well, making for a proof of concept, but it only gave me one "microstep" at a time, so the results weren't all that amazing.
It was then that it dawned on me: What if I tried to wiggle the head carriage during read attempts -- just enough for the heads to travel back and forth for the entire width of a single track, but no further? It turns out, this was surprisingly easy to accomplish, by rotating the motor from the back on 3.5" drives, and by grabbing this screw on my 5.25" drive. And the results? Absolutely mind-blowing...🤯 Granted, wiggling the heads just the right amount requires precision, as the reads will get garbled if you cross over to the adjacent tracks. To get good and consistent results, you need to do one track at a time, and set --revs=20 or thereabouts, to have enough time to move the heads without rushing, which leads to overstepping. Once I had perfected my technique, I started ploughing through my problem disk piles, salvaging close to literally everything in them, saving hundreds (if not thousands) of euros in software I no longer needed to rebuy. All it took was a few hours, and I got clean dumps out of a mountain of disks I had had no luck with, in spite of, in some cases, years of trying. For a preservationist, this was just too good to be true -- a feeling I'll probably remember for the rest of my life.
So here we are, I've named this technique The Wiggle Method, and I'm ready to accept your Nobels and other noteworthy prizes for the discovery. Hope it works out for you as well as it has for me, and feel free to ask me to expand on this section, to provide more details you may desire about the procedure.
I'll be adding pointers, caveats, and other pertinent information below:
- To figure out whether or not you're getting good reads, start out with a disk that has an existing dump out there. When looking at your microstepped track and the corresponding one in the existing dump side by side in HxC, you can clearly see when you get there: 1) the problem sector turns green, 2) the CRC values match, 3) and all of the data matches when copy-pasted to a diff checker of your choice. Be careful to check that you got all three of those things, as you can easily fall into the trap of celebrating prematurely, when for example you notice the color turns green, but it's in fact because you crossed the border to the adjacent track, where all the sectors were green to begin with, and what you're witnessing in HxC is a dump of the wrong track. It takes a bit of time to figure out all this stuff -- but once you start nailing it, you're gonna be on cloud nine.
- Don't use The Wiggle Method on your best, closest-to-factory-aligned drive, as the wiggling likely won't do it any favors. The Wigglers Association recommends preparing and reserving a secondary / tertiary / throwaway drive for wiggling purposes.☝️
Fixing bad dumps with Applesauce and HxC
What if even the (soon to be legendary) Wiggle Method isn't enough to save your disk? Often times with radial scratches in particular, you get left with a handful of bytes (hex values shown in the right-hand side of HxC) whose values are skewed due to damage, causing the affected sector(s) to fail CRC check, and your dump to be considered "garbo". (PS. why does this so often boil down to a single byte being unsalvageable -- aaarrrggghhh!!!)
When nothing you do gets a proper read of the problem spot(s), the only option left is to correct the flux information manually. Yup, that is something that is actually doable, thanks to the advanced floppy image tampering features of Applesauce for Macintosh. Let's not waste too much time rolling our eyes at the fact that we need a Mac for this -- and instead, head to the nearest recycling shop, grab an old Macbook for €20, and update it to MacOS 10.14 (like I did without issue with my mom's old 2009 Unibody, which I also upgraded with an SSD and 8GB RAM). A Core 2 Duo machine won't be useful for too much else, but having access to Applesauce will save your butt quite often, so I recommend making an effort to have it in your arsenal. (To get more speed, you can also prepare a Hackintosh install on one of your PC's, and boot into it when needed. Unfortunately, I haven't had much luck with VM's, as running the software at a usable speed would require way more graphics memory than hypervisors seem to allow for MacOS clients.)
Also notice I said we're going to fix the "flux information", and not simply rewrite the bytes of a converted .img or .d88, which is trivially easy to do with any standard hex editor. Technically speaking, HxC also comes with tools for flux editing, but the interface is a bit nebulous, and (as of this writing) it lacks the ability to output a solved flux format (like .mfi), confining us to non-flux formats for our edited files. I'll be sure to update this segment if HxC eventually reaches feature parity with AS in this regard.
At this juncture, I should probably ask: You do have an idea which specific bytes on your disk are skewed, and how? If not, the easiest way to see is to diff the problem sector from your dump and a known good one. If your disk is undumped, things get a bit trickier. HxC shows "clock violations" as tiny red bars above the giant green/orange ones representing sectors in Track view. These can apparently mean many different things, but they're a good indication of where damage is causing timings to distort. Zoom in far enough, and you start to see the data (hex values) over black vertical lines representing flux transitions. Odds are the bytes that fall under the red bars are the ones that are off. In Edit tools, you can Set start and end points around that area, and hit Repair to try brute-forcing the right value that will make the sector pass CRC check. When there are multiple spots that need fixing on the sector, you'll need to Repair all of them in turn, and hope the last one will make the sector turn green. If it doesn't, well, you can try readjusting the start and end points for each of the problem spots, but other than that, you're pretty much on your own to figure out the right values. I think Applesauce shows us a bit better view of the flux transitions themselves, which can provide some additional clues, so we might as well move on to the Mac side of things, and continue our detective work there.
Now then, are we booted up to glorious MacOS? Load up a problem image -- no, not Takuya x Ayumi💧-- and you'll be greeted with a view like this. Navigate your way to the problem spot using one of the many tools available: The Warnings tab usually has a convenient link that'll take you to the right track; once there, you may need to search for the string of bytes surrounding the problem spot, unless you spot it from the Nibble Stream. Once found, click this icon to open the Flux Stream panel. There, you'll notice each byte is composed of several "flux transitions", represented by pulses. The first ones affect the value a lot, with each subsequent one moving the value in smaller increments. The goal is to adjust the right one(s), to get the value you want. If you're able to hone in on the problem byte (using the methods outlined in the previous paragraph), the pulse that needs tweaking is usually quite easy to spot, when the damage has for example moved it too close to another one, making it stand out like a sore thumb. (Alternatively, you may need to delete an extra pulse, or add a new one, depending on the type of damage in question.) Hit enter when done, and the software will automatically see if your tinkering got the problem sector to pass CRC check.
Once you've managed to fix everything, go to File -> Export to output an .mfi file, which is a solved flux format designed for MAME, and pretty much the best (only) one for this purpose at the time of this writing. Do note that support for the format is still fairly limited -- reportedly coming Soon(tm) to Greaseweazle and HxC. In the meantime, you may have to convert it to other formats (using Applesauce) to work in many everyday use-cases.
The three vs. five revolution debate
According to the developer of HxC, there is a type of protection that can require five revolutions to be preserved properly. It is called "weak bit" protection, and here is a picture of how it looks like in the pie charts. This presents us with a bit of a conundrum, as to how we should go about dumping our disks. The crux of the matter is that, in my experience, there are mainly downsides to increasing the number of revolutions beyond three: The wear on the disk increases substantially, which may be fine in most circumstances, but will lead to catastrophes in many situations where three revolutions wouldn't have been enough to, say, scratch the disk deep enough to kill the data. Far from an academic hypothetical, when dumping a disk for the first time, even when the surface looks fine in your eyes, bad things will regularly start happening when the disk starts spinning, and the read heads start doing their thing. In these situations, the difference between three and five revolutions becomes a matter of life and death. For an example of this, see this picture, where three revolutions was just low enough for the damage to not be fatal, letting me preserve this precious disk on second try.
To make matters worse, to counter these downsides, there's not much to be gained from higher revolutions in terms of preservation fidelity, outside of preserving this one fancy protection type. That said, if I simply advocated doing three revolutions (full stop), it might lead to the proliferation of bad dumps of disks with weak bit protection. So what is the optimal workflow that takes into account all these concerns? My answer is: doing your first dump with three revolutions, and then, when spotting weak bit protection in the pie charts (or simply as a precautionary measure for all your disks, if you're not confident in telling protection schemes apart), doing a second dump with five revolutions. I know doing this much work per disk starts to get quite daunting, but when those disaster scenarios eventually start cropping up, you'll thank youself for having done the extra work, to avoid catastrophe.
Even in situations that aren't quite as dramatic, I can tell you from experience, when witnessing any type of degradation that has resulted from your dump attempt(s), it feels great to never have to think, "what if I had gone --revs=3 instead?"
The Krugman Hypothesis
Over the years of looking at the number of sectors displayed on the top left of the disk visualization (pie chart) view in HxC, I've noticed patterns that are very useful for telling apart intentional read errors (copy protection) and unintentional ones (dirt/damage). Floppy disks can be partitioned in multiple ways, and each of these produces a different total number of sectors. Examples of such "standard" numbers are 616, 640 and 2080 (on 2HD disks). (NB: Sometimes you see a number a bit above one of these, like 618, 641, 2081.) My hypothesis is, when you subtract the number of bad sectors from the total, and get a number equal to or (slightly) higher than a standard number, that information is enough to conclude you're looking at copy protection. (E.g. 618 - 2 = 616 -> the dump is good.) Conversely, when this math gets you a number below a standard number, you're looking at read issues from dirt/damage. (E.g. 640 - 4 = 636 -> the dump is bad.)
While not exactly adhering to the rigors of scientific method, it's very rare for this neat trick to give you the wrong answer. That said, there are definitely edge cases (and exceptions that prove the rule!) out there, so do let me know if you run into situations where the hypothesis fails, so I can make a list of things to watch out for. The only time I recall this happening was with a "bit flip" protection (where code on the disk tells the drive to invert a CRC value, letting it see the orange sector as green), which I've encountered in exactly this one game so far.
Realigning drives
Let me preface this section by saying I might not want to put too much effort into it right now, as according to this GitHub thread, realignment tools are coming to Greaseweazle itself, at which point the method I've been using may well become obsolete. That said, I should probably go through the steps for the benefit of people with other flux readers, at the very least.
Back in the day, realigning floppy drive heads was done using expensive precision equipment, as well as drive model-specific alignment disks, which are unfortunately impossible to reproduce using current technology, even with all the advances in floppy drive controllers and what not. So, without access to a time machine, and keys to the floppy factory (or fancy repair shop), we're going to have to do with less accurate methods -- that is, unless you're willing to fork out $$$ for an oscilloscope, and whatever eBay scalpers are asking for a genuine calibration disk for your specific drive, when once in a blue moon one of those appears for sale.
These less accurate methods consist of doing everything by hand, while being guided by a software that reports how many sectors the drive heads are able to find at their current alignment. The go-to software for this, ImageDisk (or IMD for short), requires "pure" MS-DOS to run, so the entrance fee to the party is building a "retro PC" setup. It doesn't have to be anything *too old* though, you just need to have a floppy drive connector on the motherboard. So what does that mean in terms of hardware generation? I think double drive support was phased out relatively early on, maybe around Windows XP era? But we should be fine as long as there's a connector, and the BIOS lets us select 5.25" as an option for it.
Note at this point that hooking up "J-PC" drives to a Western MS-DOS machine may introduce problems, like having to switch a bunch of jumpers on 5.25" models, to enable IBM-compatible support, and not all models have all the required jumpers accessible. If you got one of the recommended models from the beginning of the guide, this is one place where you reap the benefits.
Once you get your drive hooked up, it's time to start IMD. Press Esc to close the welcome screen, and you'll be presented with a variety of diagnostics tools, each accessible with a hotkey. A quick useful one for our purposes is the RPM test, which you start by pressing T. If your disk spins up ok, and settles on the correct RPM, that's all we need to know the drive is set up correctly. If you encounter problems at this stage, it's almost certainly due to jumper settings being incorrect in some way. Once everything is sorted, return to the main menu, and start the alignment tool by pressing A. Once your drive locks in on a track, and starts reporting the number of sectors to the software, you'll start to hear loud Beeps at short intervals, with the pitch indicating how many sectors the active head is seeing. You want to be looking at the numbers though, so unless your setup is such that you need to go at this blind, i.e. without seeing the monitor while adjusting the heads, I suggest disabling the beeps by pressing B. Now, for operating the drive, you switch the active head by pressing H, move the head carriage in increments with +/-, and jump to specific tracks with numbers 0-8.
The first thing you want to do is go through the entire disk with the bottom head (H0) set as active. The bottom head has relatively few options for adjusting its position, so we want to make sure we get perfect reads for it, before switching over to checking the top head. It's honestly been a hot minute since I had to adjust the bottom head on any of my drives, but IIRC, the way you do it is by loosening the screw that holds the carriage in place (this one on my FD-55GFR), and moving the entire carriage by hand until the bottom head sees all sectors on all tracks of the disk. Once it does, tighten the screw, and switch over to the top head (H1). The top head tends to be a lot more finicky than the bottom one, probably due to the finger-like design, which makes it so that it can become misaligned in all possible directions. The way to realign it is simple enough: loosen the screws holding it in place, just enough that you're able to move it around, and try to find a spot where it finds all sectors throughout the disk. If no position gives you perfect reads, try pressing the head down lightly with your finger, to see if it's misaligned "upward", which can happen when lifting the head too high when cleaning it, making the spring holding it down loosen too much. To counteract this, most drives come with an adjustable spring, whose "tail end" can be placed in one of several grooves located next to each other, which provide different levels of resistance. Curiously enough, one of the go-to 5.25" models, NEC FD-1155D lacks this adjustability, and requires DIY shenanigans, which I talk about in this blog post.
That's honestly pretty much all you need to fix a misaligned 5.25" drive. As for 3.5" ones, I gotta say I don't have that much experience, as my units have kept their alignment pretty well so far. One time I took one apart, and realized aligning it is a lot harder than with 5.25" drives, probably due to the smaller form-factor requiring way more finesse. I'll be sure to complement this section when I get around to trying it again, this time with more thought and effort put into it.
Until then, I hope you find the information about 5.25" drives helpful, and manage to get your units back in working order, without too much trouble.
Chapter 3: Recommended reading
Great article on extreme data recovery, outlining what "the next step" after flux is
Did you enjoy this guide? Be sure to check out my other ones too, by jumping on the main page. If you'd like to support my work, check out my cheeky Patreon (and for bigger donos, ask for my PayPal lol.) I'm low-income enough to e-beg without shame!! 🤣 Thx & TTYL!