More Mediamax site news

The missing file I was monitoring for signs of progress is now back! I'll do a more systematic test to see if the rest are too.

This is new on the official Mediamax blog. As usual, you can't leave comments there, so leave them here. Hopefully someone from Mediamax will drop by some time to address some of the points people have been leaving in comments:

http://mediamaxblog.blogspot.com/2007/08/site-update_28.html

---Mediamax blog quote

Site Update

Thanks for your feedback regarding FTP availability and MediaMax XL performance. We're listening to your concerns and in response are in the process of doubling our Engineering staff. This will enable us to respond faster to user issues.

As we've mentioned earlier, we have a completed FTP product that we are in the process of QA'ing. It's taking a while, but we need that time to ensure we release a quality product that you will enjoy using.

MediaMax XL is still a beta product. We hope to release the newest beta release shortly. We've gotten a lot of good customer feedback that will greatly enhance the usability of this product as it evolves.

Upload performance continues to improve. We had another record week in this area which is reflected in a decrease in support queries regarding uploading.

Thanks for your business and your support!
---ends

I wonder what it means when they say they are doubling engineering staff. We have learnt through this blog that less than a dozen people work at Mediamax. Take off the CEO, Customer Support manager, at least a few customer support staff (there have been a few different names appearing), accounts, security, reception etc. then 'doubling' could actually mean employing one more person!! Two if we are lucky.

Site news from Mediamax

The following is from the official Mediamax blog. See:
http://mediamaxblog.blogspot.com/2007/08/site-news.html

You can't leave comments there, so leave them here.

---quote begins

Site news

We've made a lot of progress this week and wanted to share that with you.

We're making a lot of progress in improving upload performance. Not only is uploading faster, but fewer uploads are failing. The site will now, once again, re-try your command instead of having your upload fail. As a result, we've seen a 20% increase in the number of uploads per day. There's always room for improvement, so we'll keep pushing for improved performance everyday.

Our data recovery project is moving along on schedule. This project will be completed in 5-7 weeks.

Finally, we've restored the automated granting of downloads to all accounts.

To keep users better informed on site status we'll be creating a second blog that will be dedicated to site maintenance. We'll be debuting the maintenance blog within the next two days.

Thanks for your support and business!

Finally a bit of disclosure

The following is from the official MediaMax blog (not sure why it opens 'Hi MediaMax').

Finally we learn a little of how our files have been inaccessible due to 'operator error'. The posting finally admits it will take 6-8 weeks to recover files. As you may recall I helpfully told MediaMax how to do the calculation on a past blog as it seemed to be beyond them.

It is a shame getting this information disclosed from MediaMax wastes so much time. They have already wasted so much of our time. So how about rolling on billing periods by 6 months as recompense?

TechCrunch article clarifications

Hi MediaMax,

Michael Arrington at TechCrunch recently did some good investigative reporting and released some details about MediaMax and Nirvanix that were supposed to be confidential information for several more weeks.

On August 7th, the following article was posted on TechCrunch. Now that it's public information, we wanted to post the article here, and also add some additional clarification to ease some concerns that have been raised by our customers.


Nirvanix To Challenge Amazon S3

Look for new San Diego-based Nirvanix to launch in the next few weeks. It aims to compete with Amazon's popular S3 storage web service and provide web developers another choice for online storage.

The company, which is affiliated with online storage startup MediaMax (aka Streamload), should also be announcing a $12 million round of financing in the near future. It's not clear exactly how affiliated Nirvanix and MediaMax are, and the company is keeping the specifics of the relationship quiet for now. There has clearly been a technology swap, though, and MediaMax is now using the Nirvanix service to provide the back end of its storage product. Also, former MediaMax CEO Patrick Harr is now running the show at Nirvanix (MediaMax founder Steve Iverson has retaken the CEO spot at MediaMax).

While the company is keeping quiet about the funding and exactly how it's affildiated with MediaMax, they are saying that they'll be filling some of the perceived holes in S3. Hopefully they'll be offering a service level agreement (Amazon doesn't).

MediaMax, meanwhile, has been trying to pull through a hellish technology transition (my guess is it was related to the changeover to Nirvanix) that plagued them earlier this summer. They've moved their corporate blog to a new site, and are talking openly with users about some of the problems they're facing.

Disclosure: I am an investor and on the board of directors of Omnidrive, which is also in the online storage space and can be considered a competitor to MediaMax and Nirvanix.


The article, in some instances is accurate, but it does contain some inaccuracies that need to be clarified. Streamload was split into two entities in July. The consumer online service became "MediaMax" and a second entity, Nirvanix, was founded as a new and separate company. We ("MediaMax") continue to focus on the consumer online storage space (B2C) as we have done for many years. We can't talk about Nirvanix as they are an independent company; however, we can say that we never changed over to Nirvanix, as the article suggested, and are not using their new system now.

In regards to MediaMax's problems mentioned in the TechCrunch article, this was only partially related to the spin-out process. One issue was trying to do too much, too quickly. Over the last six weeks, we completed a new round of financing and corporate spin out, moved MediaMax into a new office, migrated our web application servers to a new hosting provider, consolidated two other data centers, and have been hiring new MediaMax employees to fill in the personnel holes that were created from the split.

In addition to these major projects that were underway (and impossible to reschedule), we were still recovering from the unforeseen storage problem encountered on June 15th. The "storage problem" was a result of operator error in preparation for the data center move. There was no power surge or hacker or failed hard disk or fundamental technological problem or aliens - we just messed up. Fortunately, it was discovered fairly quickly and the data is recoverable.

If you'd like the gory details, here is exactly what happened (we have always enjoyed reading those stories about NASA probes crashing into planets because someone typed + instead of -, but didn't think it would actually be us): We wanted to manually "shrink" part of the storage system and a database that manages files in closed accounts, trash folders, etc. That is, we didn't want to move files that had already been deleted, but not actually removed from disk or the database. Normally an automated process handles this type of task, but because of the move, we thought we could create a manual process to reduce the amount of data we would have to move. And that's what messed up - this new process that should have just taken out the trash, went far beyond what we thought it was doing. Once realized, the process was stopped but significant damage had already been done. So then we started the recovery; the "shrink project" failed and we just ended up moving the entire database anyways. So the potential data loss (again, it is being recovered) was due to the most common cause for data loss: operator error.

To prevent this from ever happening again, we have put in place an even more rigorous code review process for any project that has a potential for affecting customer data like this.

So, that's the short story behind the story in the TechCrunch article. Spin off done, and data recovery is in progress. We have brought in as many additional resources as possible to help accelerate recovery, but it will still take an additional 6-8 weeks.

Over the years, maintaining the ever growing infrastructure to run MediaMax has become a distraction from our core business - to build the absolute best-of-class consumer storage applications. With this new corporate structure, we can fully focus on MediaMax.

We appreciate your support and patience and we'll continue to update you here.

Your files are coming back

Your files are coming back. John Hood of Mediamax has just posted this comment on the entry asking how long the missing files will take to process.

---
We usually do about 800GB a day in uploads when everything is running smoothly. We're obviously not back to that level yet, but are running at a few hundred GB per day.We are about a third of the way through restoring access to files. I don't have a timeline for you beyond what Steve has already stated. We have more than a petabyte of files to sift through. What I can say is that 99.9998% of the files that we have tried to restore access to have been successful. We're working with our vendor to speed up the process.
---

A petabyte is 10 to the power 15. So at 10 terabytes (10 to power 12) per day, as Steve stated, we are looking at 100 days to completion. If it has taken this long since 15 June to get through a third, that sounds about right I guess.

So now we know.

If someone wants, you could calculate the volume that is not expected to be recoverable.

OFFICIAL BLOG: Support emails

Posted at:
http://mediamaxblog.blogspot.com/2007/08/support-emails.html

---
We've heard from several customers that they are not receiving emails from our Customer Support team. We've done some internal testing in response to your queries. Our emails are frequently blocked by AOL and are almost always blocked by Comcast. To ensure a timely reply from Support, AOL and Comcast users should email us from an alternate account such as Yahoo or Hotmail.

Please don't forget to include your username.

We are currently working with both companies to resolve this situation.

Thanks for your support!
---

Comments are disabled on the official blog, so leave your comments here.

Uploads per day figure, but what about the downloads figure we want?

Well we learn something else through this blog, which has not been posted on the official one. John Hood says in a comment on the blog of 3 August: "Uploads HAVE improved since this the downtime from this transition. We are performing more than 100,000 a day."

Cool. I wonder what the official success rate is given the experiences of failed uploads people are posting here.

Another question that should be easy to answer I posed in the 3rd August posting, but got no answer. If the missing files are being recovered at the rate of 10 terabytes per day, as claimed by the re-called CEO, Steve Iverson, then how long till completion?

My dead mp3 files are still dead. Just checked. This problem arose on 15 June according to Steve, so if they have been recovering files since then, it's about 50 days or 500 terabytes already processed or 500,000,000 MBytes.

Wow! If 500,000,000 MBytes of files have come back on line, it seems to me that there must have been an awful lot of people over the past 7 weeks who have suddenly found their files available once more. If you are one of them, please let us know. It will give the rest of us some hope.

In the meantime, John, can you type the numbers into your calculator so we know how long till completion? The sum is:

Time in days = Volume of files still to be processed in terabytes/10.

Tweaking to do

Posting Sunday 5 August on Mediamax blog:

---
We've still got some tweaking to do, but upload volume remains high. We realize that some uploads are still not completing and are working on a solution. We are also still working on access to some user files. Much progress has been made in this area over the last week.

Keeping checking here for updates.

As always, thanks for your business, patience and support!
---