Files lost forever? What are the next steps?

There has been no word from Mediamax on their blog, but according to a comment posted here, Mediamax CEO Steve Iverson has said the file recovery process has completed.

Files were lost on 15 June when Mediamax botched the move to new servers. Customer services were awful communicating news on this and it was only through this blog that we learned the scale of the data loss. We were also promised substantial recovery of files.

John Hood said:
http://mediamaxusers.blogspot.com/2007/08/files-coming.html

---
We are about a third of the way through restoring access to files. I don't have a timeline for you beyond what Steve has already stated. We have more than a petabyte of files to sift through. What I can say is that 99.9998% of the files that we have tried to restore access to have been successful. We're working with our vendor to speed up the process.
---

A petabyte is 10 to the power 15.

Well, WillTRiker posted about communication with Steve stating:
http://mediamaxusers.blogspot.com/2007/10/mediamax-v6.html

"basically this is finished and according to Steve they nearly got everything back that was accidentely deleted."

Martin posted a comment on that post saying that 36% of his files are missing.

---Extract begins
File recovery. Steve Iverson says the process is finished and they got back ‘nearly everything that was accidentally deleted’. I’m not saying that Steve Iverson is lying about this although I have carefully measured that 36% (9,625 missing out of 27,107) of the files in my major collection are still showing as ‘inaccessible’ and other people are also reporting similar loss rates.

Steve is probably right they DID get back most of the files that were accidentally deleted on 15 June 2007 but they did NOT restore the several thousand files that have been affected by ‘collection rot’ over the previous years and almost certainly many of which were previously lost during the August 2006 migration from Streamload to MediaMax.

Dress it up in carefully worded half truths if you like Steve, but the net result is that MediaMax has LOST 36% of my files, and presumably affected other long term users to a similar extent. Perhaps most of these files are in fact still stored somewhere on your servers and it is just the directory links that are stuffed up. Whatever the explanation, around 250Gb of my files are gone. Actually, that probably pleases you as it is 250Gb less storage that you need to provide and goes some way to ‘clearing the dead weight on the servers’ (whether it is truly dead weight or not). Does it please me to have lost more than one third of the files painstakingly uploaded during the past four years while being continually assured that they were ‘safely and reliably’ stored by MediaMax? Well what do you think!
---extract ends

Not the 0.0002% John Hood suggested would be the case.

And no news or apology on the official blog!!

The Mediamax website still promises:

---
MediaMax, powered by Streamload, gives you a private and secure place to upload, store, access, and share your personal videos, photos, movies, music, and files.

* Store files securely on the web. Get 25 GBs of FREE online storage.
* Access your files from any web browser.
* Share all your files quickly and easily, without file size restrictions.
* Host videos, audio, images and more on the web.
* Backup your files and data safely and reliably.
---

Here is a screenshot:

Click on it for the larger version.

I haven't explored the terms and conditions, and perhaps there is a disclaimer saying that 'secure storage' means that all your files may be lost without compensation. But even if there is, it seems to me that advertising a service that the company has no track record of delivering successfully and continuing to bill customers who are not receiving that service is a little suspect.

So I'm tagging this entry with 'legal action' for anyone who is interested in pursuing this line or offering advice on it.

File availability audit 2 October 2007

This is from Martin:

My latest file collection audit made around 2 October, shows only very minor improvements. Overall 35 more files have been restored. That leaves 9625 files still missing. See graph
http://martinsotr.dnsalias.com/mediamax/Mediamax%202007.10.02.gif

Included here (click on it for a larger version):


The tiny slivers of pale green indicate the minuscule improvement made during the past week.

File availability audit 23 September 2007

The following message was sent to Mediamax and posted as a comment on an earlier blog entry by Martin Hood. I thought it would be useful to have here. I'll add a permanent link at the side so we can update progress on this entry and start tagging entries by theme:

---
An Open Letter to MediaMax
(accompanying the graph for MediaMax Files Audit #4 2007.09.23)

MediaMax File recovery progress graph http://tinyurl.com/2tuyxx
(included below).

MediaMax,

I first posted this graph 5 weeks ago to show that you WERE recovering he missing files. It was my intention that it would provide encouragement to those who doubted that the recovery was happening and would support your assertions that things were getting better. Regrettably the end result seems far removed from my hopeful optimism of 5 weeks ago.


Even the most casual look at the graph will show that it contains very disappointing news. By your own predictions, your file restoration process should be pretty much complete by now. Certainly no spectacular changes to the number of restored files would be expected at this late stage.

There was a temporary reversal in the week leading up to 16 September, when the number of missing files from those uploaded between April and July 2007 jumped from close to zero up to 30% - 40% missing. John Hood indirectly explained this in one of his blog postings by mentioning that some servers had been taken off line as part of the recovery process during this week. That also probably explained why about one third of recent uploads became inaccessible during that week. Most (but not all) of these files are now accessible again.

In the past 7 days a few minor gains have been made. The slivers of orange on the graph are the only improvements apart from the large area of orange between May and July 2007 which is just clawing back gains that were undone by the offline servers during the previous week.

Regrettably the broad summary result is very gloomy indeed.

Upload time period lost /total = %lost
2007 Jan - June 418 /3503 = 12%
2006 Jul - Dec 1009/4856 = 23%
2006 Jan - Jun 1592/4195 = 38%
2005 Jan - Dec 2242/7753 = 29%
Pre 2005 5012/7872 = 64%

Fortunately the restoration process has done a better job for more recently uploaded files with some months almost completely restored, although 12% average loss rate can hardly be called 'a complete recovery' by even the most liberal interpretation of the term.

Unfortunately the loss rate gets progressively worse the further back we go with some spectacular peaks. What did happen at Streamload in November 2005 that has caused the loss rate to remain at a whopping 91%?

Anyone who has files uploaded prior to 2005 is looking at a devastating 40% - 75% (average 64%) loss rate for these files. This is completely unforgivable for a file storage 'service' that still claims 'Store your files SECURELY on the web'. Anything above a 0% loss rate is NOT secure. 75% loss rate is patently INSECURE.

I CAN get over losing 9656 out of 27472 files (35%) that were stored on MediaMax given that local copies still exist, but I certainly cannot even consider uploading them all again. Some people who trusted you when you said their uploaded files were 'secure' have learnt a very hard lesson when they did not keep local copies.

Time and time again during the botched migration from Streamload to MediaMax in August 2006 your spin was 'Your files are safe and secure'. I know this because I kept a copy of the free-for-all Streamload blog from 22 August to 2 October 2006 and indeed the old blog pages are still on-line at http://blog.mediamax.com (archives). I cannot confirm that the files were in fact 'safe and secure' 12 months ago immediately after the migration from Streamload to MediaMax, but they certainly are not 'safe and secure' now.

You have been much less forthcoming with comments during this latest debacle and unsurprisingly, neither of the words 'safe' or 'secure' has been used anywhere in the recent MediaMax blog.

On behalf of all MediaMax users I ask the following questions:

1. Will you yet recover a substantial portion of the files that still show as missing (or indeed any of them) or are they lost irrecoverably?

2. Will you work with your user community to develop an efficient mechanism to identify lost files and provide a way of rehabilitating these files without having to upload modified copies?

3. What assurance can you give that files uploaded in the future will in fact be 'safe and secure' given that your track record to date has been woeful?

Martin Hood

Mediamax article censored - and how file restoration is progressing

In the last posting here I linked to an article posted on newsvine. The author reported that he had been forced by newsvine to enable comments on the article as Mediamax wanted a right to reply. This is ironic as Mediamax does not allow people to comment on the postings on its own blog.

All the same, the author enabled comments and many began to appear, supporting his report of lousy service from Mediamax.

As you can see from the last post, the link is now dead. Newsvine removed the post anyway. Why? Good question. One thing we do know is that Mediamax did not exercise its right to reply - perhaps fearing that people would point out any errors it may contain.

You can read the original article and others on Mediamax/Nirvanix (a Mediamax spin-off company) at:
http://nirvanix.nfshost.com/

And you - and Mediamax - are able to comment on that site.

Mediamax is not only able to comment on the Mediamaxusers blog - many of us are appealing for it to do so!!

We want some feedback. Emails from customer services are often useless - as past postings and comments here show. There has been no post on the official blog for over a week.

So this blog is the place to come for updates that actually have some bearing with reality.

Thanks to Martin Hood for his comments on the progress with missing files returning. This is his most recent:

---quote begins
It's true, the files ARE coming back!

Let me state at the outset that I am no apologist for MediaMax. I am a long term subscriber with a collection of some 30,000 audio files stored on MediaMax and like everyone else was horrified to find that almost all of these files became inaccessible on 15 June. Dismissal was very light retribution for the former employee who was responsible for this debacle!

The 30,000 files have been uploaded regularly since July 2003 and should provide a representative sample over time of the restoration of files by MediaMax.

I have performed a 100% audit of the availability of all these files on August 18 and again around August 31 and there was a marked improvement at the second sampling with the overall inacessible rate down to (only!) 45% from more than 80% a fortnight ago.

The availability improvement varies greatly with the date that the files were uploaded and this graph shows how well the file restoration is proceding for the files uploaded during the past four years.


There is still a long way to go before all files are restored and my own estimates of how long this will take are are 6-8 weeks and are consistent with those quoted by John Hood of MediaMax (no relation).

I am encouraged by the results of this research but will reserve judgement about the effectiveness of MediaMax's file recovery until the end when I see what the residual loss rate is. Even a modest 2% final loss rate will be totally unacceptable for me as this will be 600 files that need to be reloaded and 15Gb of bandwidth. With the current frustratingly unreliable uploading options, that is a huge amount of work.
---quote ends

Wow! That graph is pretty frightening. The area under it is the black hole of missing files. There seems to be no connection between date of upload and return of files. And there is an awfully long way to go.

Uploads per day figure, but what about the downloads figure we want?

Well we learn something else through this blog, which has not been posted on the official one. John Hood says in a comment on the blog of 3 August: "Uploads HAVE improved since this the downtime from this transition. We are performing more than 100,000 a day."

Cool. I wonder what the official success rate is given the experiences of failed uploads people are posting here.

Another question that should be easy to answer I posed in the 3rd August posting, but got no answer. If the missing files are being recovered at the rate of 10 terabytes per day, as claimed by the re-called CEO, Steve Iverson, then how long till completion?

My dead mp3 files are still dead. Just checked. This problem arose on 15 June according to Steve, so if they have been recovering files since then, it's about 50 days or 500 terabytes already processed or 500,000,000 MBytes.

Wow! If 500,000,000 MBytes of files have come back on line, it seems to me that there must have been an awful lot of people over the past 7 weeks who have suddenly found their files available once more. If you are one of them, please let us know. It will give the rest of us some hope.

In the meantime, John, can you type the numbers into your calculator so we know how long till completion? The sum is:

Time in days = Volume of files still to be processed in terabytes/10.

Download problems continue

Here is a round up of messages from Mediamax on download problems.

Some files are unobtainable. They are working on it, but no idea of when files missing for over a month will be obtainable. This was a reply I received from the new improved support email service (seems like you will get a reply now within a day or so).

---Mediamax support 30 July

The fix is ongoing and in progress, but I don't have an ETA on it's completion. I'm sorry I don't have a better answer for you than that.

---

I have tried to upload the missing files as I am fortunate to still have copies. Mine are mp3 files. They do not appear. This is what Mediamax support had to say.

---Mediamax support 28 July

There's an issue with some bit for bit copies, that compunds this issue, a new upload of an existing file won't work unless you change the content of the file itself. Just changing the file name won't work, you need to edit something like the metadata to change the checksum of the file. A new upload should then result in a working file.

We are very sorry about this and are working to resolve it but please try the above workaround in the meantime.
---

I responded that changing an mp3 file is not a particularly easy workaround and my preferred approach is to use another service.

Now, some people are unable to download because their download limit has been reached and not re-credited at the start of the next period. This is the response from Mediamax:

---Mediamax blog 30 July 2007

Download Credits

Some users with accounts renewing the last four days of this month may not have automatically received their regular allotment of downloads. If you are one of the users affected please email us at support@mediamax.com. In your subject line please type "Download Credits". We'll have your download credits posted within 24 hours of receiving your email.

We are working on a solution to prevent this from happening next month. We apologize for any inconvenience.

Thanks for your patience and your business!

The MediaMax Customer Support Team

----

Finally, work is underway for an FTP system to upload and download files. There is a suggestion that Mediamax may actually get this working before subjecting us to it. This is the Mediamax blog 28 July.

----Mediamax blog 28 July 2007

FTP Update

We've gotten inquiries from users about the return of FTP uploads. We have a new version that will allow the management of both uploads and downloads. (The previous version only allowed for the management of uploads.) This added functionality was a request from many of our users. We're currently performing quality assurance testing on the new version to ensure full functionality. We'll post here when it becomes available for use.

---

So still not much joy for customers.

Welcome Mr. John Hood from Mediamax

I would like to formally welcome Mr. John Hood, Director of Customer Support at Mediamax to this blog. I have been alerting Mediamax to its existance since setting it up, so nice for someone to drop by.

This is Mr. Hood's reply to comments on an earlier entry about the lack of response to emails:
http://mediamaxusers.blogspot.com/2007/07/lost-emails.html

---
Nothing could be further from the truth. We are most definitely listening (and reading...)
We were swamped with emails which slowed down response times as I said in my post. We've got a better ERM system now and are up to the challenge. Please shoot us an email.

John Hood
Director of Customer Support
MediaMax
---

I'd like to invite Mr. Hood to respond to some of the other key concerns people have been raising here. Specifically:

1. When can we expect files that no longer stream or download to be accessible again?

2. As there appear to have been breakdowns in service periodically since Mediamax took over Streamload, how can we have any confidence that there won't be more? Some users have lost access to files for the past month. This is not a minor issue.

3. What compensation will be offered to users? How about extending the billing period by twice the down time? Two months extension owed so far. Those off us who have had to go to extra expense and trouble to put services like podcasts back on air would perhaps see this as getting off lightly.

4. When will refunds be made to people who have been billed after closing their accounts?

Feel free to leave answers in a comment or I can give you posting rights if you contact me.

Alternatively why not answer these questions on the Mediamax blog and allow people to comment there?

Are your mp3 streams not working?

My older mp3 stream on Mediamax have not been working since this move to the new system last month. They are still not working today.

However, since a few hours ago I have been able upload files and they appear soon after, if not immediately. Hurrah.

So I though I would try to sort out my dead mp3 files. The bad news is, the ones on the server seem unusable. Worse news is, that uploading the original file doesn't solve it. But I have found what seems to work. If you don't have a back up of your file though, you may be lost. And if you were using Mediamax to back up files then I feel for you! From the comments left here, it is looking like people have lost some important stuff. Let us hope Mediamax will recover it.

Okay, the mp3 problem. My files are still there in the 'hosted' folder. But when I try to access them through the Mediamax control panel it just sits there forever, waiting for the server. If I try to access them via the link on my blog it's as if there is nothing to download.

I still had a copy of some of the files in my 'filemanager' so replaced the 'hosted' file with that. Made no difference. Tried downloading the file and got an error message. So it looks like it's been corrupted.

I uploaded the file again and copied it into 'hosted'. Wouldn't play. Deleted the originals, uploaded again. Still wouldn't play. Plays fine on my computer.

Then I remembered seeing (in the blog I think) that if you upload a duplicate file, this is recognised and so instead of taking up space a link to the existing file is made. The tip I read said you have to make a change to the file if you want a separate file - not the name, but the contents.

So I cut a bit of silence from the file and uploaded again. Now it works. Almighty pain to have to modify each file, but as they are podcasts I can do that. If it was music, would be difficult, I imagine. I can't find the blog about this at present, but have found this one, which implies the same thing:

http://blog.mediamax.com/?p=25
---
...in addition to an industry-leading 250 GB of unique storage, we will also not count duplicate files stored on our servers against that 250 GB storage limit. So, if many Streamload customers are storing the same Britney Spears MP3s that you are storing (come on, admit it!) those MP3s won’t count against your storage quota.
---

And, I guess, if the file gets corrupted, it's corrupted for everybody.

Perhaps the deleted files will really be deleted during system cleanup and it will be possible to upload a file without having to change its contents. As it is, it's a pretty awful work around.

Some Mediamax streams still down and upload problems

Mediamax/Streamload streams went off air on 25 June with a major maintenance programme of work. According to the Streamload blog, the new server was on line on 1 July.

However, there are still problems, as acknowledged on a Streamload blog on 2 July. That's included below. The comments are deactivated on the Streamload blog, so leave comments here.

We don't want to distract the techies from solving the problems, but they have also asked people to sent in reports to support@streamload.com

I've had most of the problems described in the 2 July Streamload blog. What is causing me particular problems still is that older streams are still not working - very frustrating as I needed them for a meeting in Second Life at the beginning of the week.

I also use Mediamax for hosting a free podcast I do, which is on iTunes. The file I uploaded for the 2 July episode has still not appeared in my directory to be able to move it to the 'hosted' folder. Same thing happened with the previous podcast, though that appeared a day or two later.

My listeners have been sending me complaints and I'm not sure how long I can go on waiting for Mediamax to sort this out. I don't really want to move to another host though - my experience so far has not been so bad.

This is from the Streamload blog, showing the type of problems they are already aware of:

---

The transfer of our IT/DB operations to an outside vendor in Chicago is complete. Of course, when a move this large occurs there are going to be a few random issues to iron out. The known issues are listed below. We are working around the clock in San Diego and Chicago to correct the situation. We will post updates as issues are resolved. Please direct any questions or concerns to support@streamload.com.

1. When trying to log into your account you might receive one of the following errors:

HTTP 400 bad request

Bad Request (Invalid Hostname)

2. When clicking on “go to account” from welcome page you may get the following error:

Page Not FoundOops! Something went wrong… Please refresh your browser window.

3. You may get a “red x” for some of the greyed buttons (download, tag, etc)

4. When clicking between tabs you may get an HTTP 400 bad request error.

5. On log-in you may need to refresh the page a few times to get to your account.

6. Uploads are processing slower than usual.

7. Some uploads time out midway through the process.

8. Some files may be temporarily unavailable.

---