ThimbleCrash
Apr 11, 2016
A few days ago, I stumbled into my home office, a bowl of oatmeal in hand, getting ready for a quick check of the Twitters before my morning run, but oddly, my computer was off. I leave my Mac running all the time and it was strange that it didn't just wake up from sleep. I powered it on and everything seemed normal.
The machine will sometimes restart in the middle of the night, and when it reboots, there is a nice message box telling me that it crashed and kindly shows me the logs. This morning, no such message enthusiastically greeted me.
Odd.
The next day I was editing a very large Photoshop files -- touching on 4GB -- when it kept popping up these errors when I tried to save saying I didn't have the correct permissions to save.
Odd.
The next morning, I headed into the home office again, pre-run oatmeal in hand, and sat down to read the emails. Most of the new email that arrived during the night had no sender or subject.
Odd.
A few seconds later, a message box pops up, asking me to enter my iCloud password, I hit cancel and switched to my browser and pulled up Twitter and then Chrome asked me for my Twitter password and had me logged out. I went to another site, and I had also been logged out and it was asking for my password again, then my email program asked me for my password, I entered it and hit OK, then a new message box come on saying the login group of keychain was missing and did I want to reset it.
Odd.
Something was going wrong and I decide to just reboot and see if things were magically fixed, because, you know, that might happen. Right?
As the machine was shutting down, it dawned on me that rebooting my machine when it was telling me the login keychain was missing might not have been the smartest idea, and I was right.
Half way through the boot process, the machine just shut down. Three more attempts with the same results. I booted in verbose mode and watched the boot process, everything was normal until it got to the disk check, then it displayed a slew of errors and shutdown.
Crap.
I booted in recovery mode and ran Disk Utility and checked the disk, sure enough, there were a crap-ton™ of missing block error messages. No problem, I'll just hit "Repair Disk" and be up and running again.
Nope.
Repair Disk informed me that it was unable to repair the disk. I was somewhat disappointed the Mac didn't emit a mechanical mocking laugh at this point.
I didn't have a Thunderbolt Cable, so I couldn't connect my iMac to my laptop and see if the drive was still readable.
I'm pretty religious about backing stuff up. Time Machine runs every hours and skips only my large video and audio files. It doesn't back up my projects and source code, but they are all in Git. I had made a few changes the previous day and I had not pushed, but it was just a few lines of code, easily retyped. The big thing I didn't have was backup of was my Windows VM.
Without it I can't do windows build for testing. Nothing of importance is on the VM except a install of Visual Studio. The VM could be rebuilt in an afternoon, so losing it wasn't climatic, just a pain. The one thing I was going to lose by doing a reformat was the podcast we did on Friday. I hadn't edit it yet, so it hadn't been archived for future generations to enjoy.
After a little more thinking about what might be on the machine and not backed up, I decided to reformat and reinstall from my Time Machine backup.
There was a very real possibility that this was a hardware problem and reformatting wasn't going to save the day. In that case, I was going to have to send the machine out to get the drive replaced and that would take several days, if not a week if I wanted Apple to do it under Apple Care. With PAX looming in a few weeks, that was not an event I welcomed.
I can do just about everything on my laptop, except make Windows builds, so it wasn't a catastrophe, just a big pain as half of our testing staff is Windows only.
I went into Disk Utility again and selected the volume, paused for a few seconds to contemplate the destructive nature of my next move, then hit Erase. A few moments later, I was informed that my drive could not be reformatted.
My iMac has what's called a fusion drive. There is a 128MB SSD drive and a 3TB spinning drive that are fused into one big drive. The OS is smart enough to move files you don't access very often to the slow spinning disk, keeping the files you need on the spiffy fast SSD drive. It's a great idea. The new macs have it, and it's been embedded in a lot of standalone drives, and there is a version for Windows machines.
The problem with fusion drives is you now have now have two points of failure. If either drive goes bad, you lose the data on both drives, which is what happened to me.
Apparently, while the iMac is happy to have a fusion drive, Disk Utility has not caught up yet, and there is no way to reformat it.
Gads.
At this point, I'm kind of stuck. All I want to do is reformat the drive and start over. Visions of days waiting to speak to Apple and weeks of waiting to get my machine back are dancing through my head, all while PAX stalks closer and closer.
I call up the local Apple store and see when I can get an appointment to visit the Genius Bar. I don't have a lot of faith in the Genius Bar to help with this issue. Normally it is filled with people trying to figure out how to get email on their iPhones. I imagine I'll bring the computer in and the "genius" behind the "bar" will shrug and tell me they need to send it in and there will be a two week wait, but if I'm having trouble getting email on my iPhone, they'd be happy to help.
I place the call and much to my surprise, they have a free slot at 4:45 that afternoon. Great. I pack the computer up and haul it in.
As expected, there are about 30 people being helped at the Genius Bar and other than a few laptops, they are all iPhones. I plop my giant iMac on the counter and wait, feeling quite out of place.
At 4:50 a nice person comes over and asks what the problem is. I tell him the machine won't boot due to a disk error. He then proceeds to talk to me like I'm a 4 year old, explaining that a hard disk has this spinny thing in them and sometimes those can go bad.
Seriously.
I then tell him I'm a Mac developer (I probably rolled my eyes), at which point he actually seems relieved and switches to full on nerd mode. He plugs my machine into the store network and boots from there, then proceeds to run some fancy diagnostic stuff I don't have access to. The good news is he doesn't find anything physically wrong with the drive.
He connects the iMac to my laptop and we mount it as a external drive. Everything seems to still be there, so I spend the next half hour copying the Windows VM and the podcast to my laptop and we reformat the machine using a bunch of shell commands, while he's happy to explain what is happening.
I ask why Disk Utility can't just reformat the drive. He says the Apple Utilities haven't caught up to the fusion drives and (politely) expresses some amount of frustration at this fact. I get the impression he's done this a lot.
I pack up the newly reformatted machine and head home. Time Machine restored perfectly, I then pulled all the repos from git, and other than needing to reenter all my passwords, the machine is back like nothing happened.
I know you hear this a lot, but back up your shit. This story would not have had a happy ending if I didn't back up everything obsessively. I run Time Machine for local backups and use Arq to keep offsite archives on amazon's S3 storage (Time Machine can't help you if your house burns down).
I did manage to restore Friday's podcast recording, so I'll try and have that edited and up tomorrow.
I lost a day, but I got a nice clean desk out of it, so I'll call that a win.
- Ron
What actually caused your headache?
Maybe run a long memory check and check the SMART values of both SSD and HD for suspicious errors.
This guide lays out the process fairly well: http://www.macworld.com/article/2014011/storage-drives/how-to-make-your-own-fusion-drive.html
I use Carbon Copy Cloner to copy my whole drive to an external one. You have a bootable copy than and if your internal drive fails you can just boot from the external one.
Cheers!
Anyway, if you weren't able to restore the latest Friday podcast, I believe it would have been published somehow by your 'fake' counterpart! :-) :-)
Glad everything ended well.
Like Ron said:
Its major disadvantage is that the propability of failure of this fusion drive is the propability of failure of the HD added to the one of the SSD, like a RAID0 does. Its goal is high performance and not high availability.
So to paraphrase an old Sierra adventure game strategy: Backup early, backup often!
In a not too distant future, HDs will be completely replaced by SSDs in desktop computers (like it is now in notebooks). Thereby also rendering the fusion drive concept obsolete.
Wait... can't you just walk into the middle of the Apple store and loudly announce, "I AM RON GILBERT, CREATOR OF MONKEY ISLAND AND MANIAC MANSION... MY COMPUTER IS BROKEN AND THUS THE FUTURE OF HIGH QUALITY POINT AND CLICK ADVENTURE GAMES HANGS IN THE BALANCE UNTIL SUCH A TIME THAT MY COMPUTER IS WORKING AGAIN"
And everyone looks around at each other with panicked faces and some of the women folk begin crying and then a big alarm siren goes off and there is the sound of helicopters as Apple special ops roll out to fix your computer.
"He is Ron Gilbert, first of his name, father of point-and-click adventures, scourge of text parsers, lord of monkey wrenches... and Protector of the Realms of Mac, PC and Linux."
https://www.youtube.com/watch?v=ldDxDHHGB6A
Endless Replay-Ability.. True to the genre.. MOD if you want.. GAME
Fuck this season pass bullshit!
Planned Obsolescence with DLC and Future Unlocked Content.. I fear for this generation Ron, I really do!
Terrible Toy box for life.
Sadly when it's all ssd I'll have to buy real coffee coasters.
But yeah data you don't back up is data you don't care about.
I'll be doing another back up later today ;)
I'm glad everything worked out better than you expected!
http://backtothefuture.wikia.com/wiki/Mr._Fusion
I am now worried about what happens when his fusion drive crashes...
Thank you
Truly these are the Aesop fables of this generation to live by.
Amazing stuff.
(And to Mark of course who shared his pixelated insights)
I've observed the abdication of dithering with mixed feelings. On the one hand, dithering is very cool, because it reminds you of the limited color palette you had to deal with back in the 8-bit era. On the other hand, I completely understand that it would be too time-consuming. Anyway, Mark's contribution to the game is unpayable, even if there was no dithering at all!
It would be nice if the upcoming graphics tool, which Mark mentioned at 8:00 and 1:20:00 in the video, was going to provide some reasonable functions that made dithering significantly easier.
"The Abdication of Dithering" would be a great book title- too bad that blog post is closed.
Besides, there is still some dithering in particular areas in TP, such as the sky. I think it's a great middle course between nostalgia, aesthetics and economic pragmatism.
Though, whenever a surface needs to appear grained, a modest use of dithering is still a reasonable stylistic option.
Wait a sec... is that a C64 in the background behind Delores? Does she back-up religiously?
I loved the part where the guy at the genius bar was giving you the kiddie treatment. I did not expect him to switch to real-useful-tech-mode, though. I would have thought you would walk away with a broken mac but now being able to send mail using your iPhone!
Sorry man, but I think uou put too much anxiety in it. The odds that 6 HDs break simultaneously are soooo low.
I have two backups. The odds I lose 3 copies of my data at once are lower
Than the odds I will die tomorrow. So I don't matter to make more copies than this. But if 6 copies can make you feel better and more confortable... Good for you :-)
What if you discover you have accidentally deleted/modified important data ? That's why you need recent and preferably multiple generations backups or use a file systems storing multiple versions of files. (Btw. this can happen due to your fault or some other persons fault like kids, it could also happen due to computer crashes etc.)
What if some virus encrypts all your data? That's why you should have backups on mediums not always connected to your computer or at least a backup generation before such incident.
What if someone breaks into your house and steals your stuff or your house burns down? That's why you need off-site backups.
Some people prefer having more recent backups and more generations due to some of the reasons listed above. Also it's cheaper to lose just 15 minutes of work instead of e.g. 1 day of work or 1 week of work. And some data is completely irreplaceable, e.g. if you lose baby pictures from 5 years ago you can never bring them back.
So: Backup early, backup often. And also have off-site backups.
With one big difference: he still got his data...
No, the only important thing is that when I'll lose all my three copies, I'll be probably dead (still a matter of odds), so I won't mind.
I have one on-site backup and one off-site backup which is closed inside a safe. This is what MY peace of mind needs, and I think that if somebody needs 6 copies, should do 6 copies. The only thing that matters is to feel good and safe.
I think I wouldn't feel good with 6 copies. Too much work, too many things to think of. Life is beautiful. :-)
But much more important:
Is that a Commodore 64 in the Top pic? Will it be in the game? Operational? Will it run a game?? *HINT,HINT*
:-)
I even think its a Commodore 64. And I hope it will run a litle Game like in Day of the Tentacle. :-)
Allowing to hook up external programs this way would be a nice alternative, e.g. setting a commandline to ScummVM in a configuration file.
I enjoyed most the trials of the Monk.
I played a few hours yesterday and the day before using the Monk, the spooky Twins and the Hillbillie. And now I'm stuck at this carnival... how appropriate!
I do miss a real save & load game feature... it seems I missed one blinking icon to reveal the first part of the back story, of the Twins and I cannot climb back up.And thanks to SteamPlay (TM), my progress gets synced over all devices (argh).
Anyone knows what the option "drop kid" does?
Here it is! The second part of the "Answers to Unanswered Questions of Friday Podcast" is online!
The 'fake' Ron Gilbert and the 'fake' David Fox will ansers another 26-pack Questions, in a ...plausible way :-)
Enjoy!
http://www.cinemapioxi.it/zak/Thimbleweed_Park_Fake_Podcasts.html
I'm sorry that I couldn't find your name in the Friday Questions post, otherwise I would have answered you in a musical way ;-)
Btw. real Ron Gilbert tried to copy you, they wanted to sing Happy Birthday (but then they wandered from the subject...)
I tell you. If it fails in the first steps means that any app is blockibg it somehow. Close all. Reopen the apps you opened since last time you plugged it. And exit all of them. Sometimes they run backups.
And then you can repair it. Works for me all the times!
I'm sorry to hear about your computer troubles, that's never a good thing to happen when you trust the electronic gods with your important work and archival materials. I had the same thing happen to me various times throughout my life. At one time I had to spend almost $3,000.00 to recover data from a dead HDD.
More recently, I encountered a severe disk problem, and like you, didn't notice it until it was too late. In attempting to recover from TimeMachine, I ended up replacing some of the usable backup with the corrupted data from the disk, rendering the backup questionable.
There are a few things that saved my bacon that day: A TimeMachine backup (like yours), a brand new HDD, and the DiskWarrior and TechTool Pro software.
I learned that day that DiskUtility is a bit lame in the recovery department: the most it can do is repair permissions and recover the "Directory" file from an older copy. This is similar in nature to what was called the "File Allocation Table" (FAT) in the DOS world. However, DiskUtility cannot mark bad blocks, nor recover data in any other way.
Enter DiskWarrior and Tech Tool Pro, which do all the rest. DiskWarrior can find and repair corrupted files (or attempt to), it can also do the same as DiskUtility, but rather than just replacing the "Directory," it can give you a comparison view (like "diff") so that you can see what is actually been recovered and maybe learn what may be lost or corrected. It also provides log files of all this, so that you may do integrity checks later if you want to (which I did).
Tech Tool Pro is a general testing utility for Macs that provides not only some rather extensive tests for HDDs, but for all hardware, include memory, video hardware, etc. It also does a surface scan and identifies back blocks, tries to recover data, and marks them for future avoidance.
I just thought to provide some additional information on disk recovery methods, since I went through similar pains as you, and had to stumble in the dark to figure out what to do and how to avoid it in the future.
Regards,
-dZ.
(To err is human, but to really foul things up you need a computer.
- Paul R. Ehrlich)
the Hillbilly
the Knight (again)
the Monk (again)
Then, finally, my favourite puzzles are with :
the Monk
the Scientist
the Hillbilly
My favourite scenarios are:
the Scientist (1st prize!)
the Twins
the island
Favourite extra characters:
Spanky & the Hermit
The Cave
The Hotdogs
Favourite playable characters:
the Twins
Yes, yes, I know: you've been all waiting this comment for years.
Btw., so much for playing one playthrough and letting the game rest for some time :-)
(DISCLAIMER: always drink with moderation; don't drink at all if you are going to drive; wine is just a drink, it will not solve your problems)
1) the Carnival, the Island with the Hermit: they are present both in Monkey1+2 and in TheCave: coincidence, quoting, or particularly evocative?
2) The Talking Cave: was it difficult to cast the voice actor? His work I appreciated a lot.
Thank you
I had a similar problem about 9 months ago, except without Fusion Drive. I have an older Mac Pro and at one point I decided to add an SSD to it but the largest SSD sold at the time (500GB) couldn't fit all of the data on the standard HD that it was replacing. I thought about Fusion Drive, and eventually decided to just manage the space myself so it would be easier to maintain a Bootcamp partition. I installed the base OS to the SSD, as well as the Applications folder, but I symlinked the entire /Users/ folder onto the old drive since that was where the majority of my large files (and Windows VM, which I don't back up either) live.
Around the middle of 2015, I had a RAM issue that caused my machine to reboot randomly and deactivate the 'bad' memory. It actually took me a few months to realize what was happening, because the performance wasn't really impacted by a lack of RAM thanks to the SSD. When I finally figured out there was a problem, I cleaned all of the dust out of the machine and cleaned the RAM sticks with rubbing alcohol. The RAM has been perfectly fine since then. But I was still having stability issues, which turned out to be caused by file corruption due to the random reboots. Both drives had issues. Repairing the SSD with Disk Utility caused the machine to become unbootable. I had to reinstall the OS to fix that and everything was fine there again, but the regular HD could not be repaired by Disk Utility.
At that point, I decided since I had Time Machine backups I would just reformat both drives and start fresh, restoring my data from backups. Then I made the mistake of scanning the Time Machine drive with Disk Utility. It decided there was corruption there too, and it unmounted the drive to repair it. But there wasn't enough free space on the drive for Disk Utility to repair it. Which is how it's supposed to work, since Time Machine automatically makes additional backups until there isn't space anymore, and clears the older ones when it begins to need space for newer ones. So on most time machine drives that have been in use for a while, there's never going to be enough free space on them to repair them. OS X would only let me remount the Time Machine drive as read only, and with the unrepairable problems I wasn't even guaranteed that all of my backups were good.
Since both my HD and the backup drive had unrepairable corruption, I didn't want to risk a reformat at that point. I had a copy of TechTool Pro that I got with some kind of software bundle years ago, I tried that but even TechTool couldn't repair the drives. Some forum posts online suggested DataResuce, but that appeared to require restoring to another drive and I didn't really want to go through the hassle of restoring over 500GB of data to another drive. I saw a lot of positive comments about DiskWarrior, which wasn't cheap, but I bought that hoping it would work. Worst case scenario, I could've just returned it for a refund. Surprisingly, DiskWarrior was able to repair both my HD as well as the Time Machine drive that Disk Utility refused to repair because there wasn't enough space. For the Time Machine drive, it gave me a warning saying that there wasn't enough free space to write a backup file table to the disk somewhere else and it would need to overwrite the existing one and data could be lost forever if the power went out, but everything worked perfectly and the drive was 100% valid after that point even according to Disk Utility. (It actually generates a brand new filesystem table to overwrite the original one, instead of trying to figure out how to salvage the existing one. I thought that approach was kind of interesting, because I'm not aware of any Windows utility for repairing NTFS drives that takes the same approach. On the other hand, I've never ran into a windows drive that couldn't be repaired by the built in chkdsk command.)
Some lessons I learned:
- Pretty much any spontaneous reboot causes some kind of problem that needs to be repaired, at least on standard HD's. This doesn't seem to happen very often for the SSD's though.
- Time Machine backup drives basically can't ever be repaired by Disk Utility, because they'll always be too full to do so.
- Splitting data across two drives is complicated when something goes wrong. I ended up buying a 1TB SSD and moving all of my primary OS data to a single drive again.
- I was still having occasional stability issues on the new drive. Running beta builds of OS X had added some random stability issues to my machine that wouldn't go away from a fresh install if I restored from backups. The only way I found to fix my problems was to do a fresh install without letting OS X restore backups from Time Machine. Instead I just copied my files back manually after a fresh install to get rid of whatever lingering preferences or profile data was causing my problems.
I guess my point of my story is to make sure you're also checking your backup drives for errors too.