Tuesday, August 28, 2007

More Mediamax site news

The missing file I was monitoring for signs of progress is now back! I'll do a more systematic test to see if the rest are too.

This is new on the official Mediamax blog. As usual, you can't leave comments there, so leave them here. Hopefully someone from Mediamax will drop by some time to address some of the points people have been leaving in comments:

http://mediamaxblog.blogspot.com/2007/08/site-update_28.html

---Mediamax blog quote

Site Update

Thanks for your feedback regarding FTP availability and MediaMax XL performance. We're listening to your concerns and in response are in the process of doubling our Engineering staff. This will enable us to respond faster to user issues.

As we've mentioned earlier, we have a completed FTP product that we are in the process of QA'ing. It's taking a while, but we need that time to ensure we release a quality product that you will enjoy using.

MediaMax XL is still a beta product. We hope to release the newest beta release shortly. We've gotten a lot of good customer feedback that will greatly enhance the usability of this product as it evolves.

Upload performance continues to improve. We had another record week in this area which is reflected in a decrease in support queries regarding uploading.

Thanks for your business and your support!
---ends

I wonder what it means when they say they are doubling engineering staff. We have learnt through this blog that less than a dozen people work at Mediamax. Take off the CEO, Customer Support manager, at least a few customer support staff (there have been a few different names appearing), accounts, security, reception etc. then 'doubling' could actually mean employing one more person!! Two if we are lucky.

Wednesday, August 22, 2007

Site news from Mediamax

The following is from the official Mediamax blog. See:
http://mediamaxblog.blogspot.com/2007/08/site-news.html

You can't leave comments there, so leave them here.

---quote begins

Site news

We've made a lot of progress this week and wanted to share that with you.

We're making a lot of progress in improving upload performance. Not only is uploading faster, but fewer uploads are failing. The site will now, once again, re-try your command instead of having your upload fail. As a result, we've seen a 20% increase in the number of uploads per day. There's always room for improvement, so we'll keep pushing for improved performance everyday.

Our data recovery project is moving along on schedule. This project will be completed in 5-7 weeks.

Finally, we've restored the automated granting of downloads to all accounts.

To keep users better informed on site status we'll be creating a second blog that will be dedicated to site maintenance. We'll be debuting the maintenance blog within the next two days.

Thanks for your support and business!

Tuesday, August 14, 2007

Finally a bit of disclosure

The following is from the official MediaMax blog (not sure why it opens 'Hi MediaMax').

Finally we learn a little of how our files have been inaccessible due to 'operator error'. The posting finally admits it will take 6-8 weeks to recover files. As you may recall I helpfully told MediaMax how to do the calculation on a past blog as it seemed to be beyond them.

It is a shame getting this information disclosed from MediaMax wastes so much time. They have already wasted so much of our time. So how about rolling on billing periods by 6 months as recompense?

TechCrunch article clarifications

Hi MediaMax,

Michael Arrington at TechCrunch recently did some good investigative reporting and released some details about MediaMax and Nirvanix that were supposed to be confidential information for several more weeks.

On August 7th, the following article was posted on TechCrunch. Now that it's public information, we wanted to post the article here, and also add some additional clarification to ease some concerns that have been raised by our customers.


Nirvanix To Challenge Amazon S3

Look for new San Diego-based Nirvanix to launch in the next few weeks. It aims to compete with Amazon's popular S3 storage web service and provide web developers another choice for online storage.

The company, which is affiliated with online storage startup MediaMax (aka Streamload), should also be announcing a $12 million round of financing in the near future. It's not clear exactly how affiliated Nirvanix and MediaMax are, and the company is keeping the specifics of the relationship quiet for now. There has clearly been a technology swap, though, and MediaMax is now using the Nirvanix service to provide the back end of its storage product. Also, former MediaMax CEO Patrick Harr is now running the show at Nirvanix (MediaMax founder Steve Iverson has retaken the CEO spot at MediaMax).

While the company is keeping quiet about the funding and exactly how it's affildiated with MediaMax, they are saying that they'll be filling some of the perceived holes in S3. Hopefully they'll be offering a service level agreement (Amazon doesn't).

MediaMax, meanwhile, has been trying to pull through a hellish technology transition (my guess is it was related to the changeover to Nirvanix) that plagued them earlier this summer. They've moved their corporate blog to a new site, and are talking openly with users about some of the problems they're facing.

Disclosure: I am an investor and on the board of directors of Omnidrive, which is also in the online storage space and can be considered a competitor to MediaMax and Nirvanix.



The article, in some instances is accurate, but it does contain some inaccuracies that need to be clarified. Streamload was split into two entities in July. The consumer online service became "MediaMax" and a second entity, Nirvanix, was founded as a new and separate company. We ("MediaMax") continue to focus on the consumer online storage space (B2C) as we have done for many years. We can't talk about Nirvanix as they are an independent company; however, we can say that we never changed over to Nirvanix, as the article suggested, and are not using their new system now.

In regards to MediaMax's problems mentioned in the TechCrunch article, this was only partially related to the spin-out process. One issue was trying to do too much, too quickly. Over the last six weeks, we completed a new round of financing and corporate spin out, moved MediaMax into a new office, migrated our web application servers to a new hosting provider, consolidated two other data centers, and have been hiring new MediaMax employees to fill in the personnel holes that were created from the split.

In addition to these major projects that were underway (and impossible to reschedule), we were still recovering from the unforeseen storage problem encountered on June 15th. The "storage problem" was a result of operator error in preparation for the data center move. There was no power surge or hacker or failed hard disk or fundamental technological problem or aliens - we just messed up. Fortunately, it was discovered fairly quickly and the data is recoverable.

If you'd like the gory details, here is exactly what happened (we have always enjoyed reading those stories about NASA probes crashing into planets because someone typed + instead of -, but didn't think it would actually be us): We wanted to manually "shrink" part of the storage system and a database that manages files in closed accounts, trash folders, etc. That is, we didn't want to move files that had already been deleted, but not actually removed from disk or the database. Normally an automated process handles this type of task, but because of the move, we thought we could create a manual process to reduce the amount of data we would have to move. And that's what messed up - this new process that should have just taken out the trash, went far beyond what we thought it was doing. Once realized, the process was stopped but significant damage had already been done. So then we started the recovery; the "shrink project" failed and we just ended up moving the entire database anyways. So the potential data loss (again, it is being recovered) was due to the most common cause for data loss: operator error.

To prevent this from ever happening again, we have put in place an even more rigorous code review process for any project that has a potential for affecting customer data like this.

So, that's the short story behind the story in the TechCrunch article. Spin off done, and data recovery is in progress. We have brought in as many additional resources as possible to help accelerate recovery, but it will still take an additional 6-8 weeks.

Over the years, maintaining the ever growing infrastructure to run MediaMax has become a distraction from our core business - to build the absolute best-of-class consumer storage applications. With this new corporate structure, we can fully focus on MediaMax.

We appreciate your support and patience and we'll continue to update you here.

Wednesday, August 8, 2007

Your files are coming back

Your files are coming back. John Hood of Mediamax has just posted this comment on the entry asking how long the missing files will take to process.

---
We usually do about 800GB a day in uploads when everything is running smoothly. We're obviously not back to that level yet, but are running at a few hundred GB per day.We are about a third of the way through restoring access to files. I don't have a timeline for you beyond what Steve has already stated. We have more than a petabyte of files to sift through. What I can say is that 99.9998% of the files that we have tried to restore access to have been successful. We're working with our vendor to speed up the process.
---

A petabyte is 10 to the power 15. So at 10 terabytes (10 to power 12) per day, as Steve stated, we are looking at 100 days to completion. If it has taken this long since 15 June to get through a third, that sounds about right I guess.

So now we know.

If someone wants, you could calculate the volume that is not expected to be recoverable.

Tuesday, August 7, 2007

OFFICIAL BLOG: Support emails

Posted at:
http://mediamaxblog.blogspot.com/2007/08/support-emails.html

---
We've heard from several customers that they are not receiving emails from our Customer Support team. We've done some internal testing in response to your queries. Our emails are frequently blocked by AOL and are almost always blocked by Comcast. To ensure a timely reply from Support, AOL and Comcast users should email us from an alternate account such as Yahoo or Hotmail.

Please don't forget to include your username.

We are currently working with both companies to resolve this situation.

Thanks for your support!
---

Comments are disabled on the official blog, so leave your comments here.

Monday, August 6, 2007

Uploads per day figure, but what about the downloads figure we want?

Well we learn something else through this blog, which has not been posted on the official one. John Hood says in a comment on the blog of 3 August: "Uploads HAVE improved since this the downtime from this transition. We are performing more than 100,000 a day."

Cool. I wonder what the official success rate is given the experiences of failed uploads people are posting here.

Another question that should be easy to answer I posed in the 3rd August posting, but got no answer. If the missing files are being recovered at the rate of 10 terabytes per day, as claimed by the re-called CEO, Steve Iverson, then how long till completion?

My dead mp3 files are still dead. Just checked. This problem arose on 15 June according to Steve, so if they have been recovering files since then, it's about 50 days or 500 terabytes already processed or 500,000,000 MBytes.

Wow! If 500,000,000 MBytes of files have come back on line, it seems to me that there must have been an awful lot of people over the past 7 weeks who have suddenly found their files available once more. If you are one of them, please let us know. It will give the rest of us some hope.

In the meantime, John, can you type the numbers into your calculator so we know how long till completion? The sum is:

Time in days = Volume of files still to be processed in terabytes/10.

Tweaking to do

Posting Sunday 5 August on Mediamax blog:

---
We've still got some tweaking to do, but upload volume remains high. We realize that some uploads are still not completing and are working on a solution. We are also still working on access to some user files. Much progress has been made in this area over the last week.

Keeping checking here for updates.

As always, thanks for your business, patience and support!
---

Friday, August 3, 2007

Is there a Mediamax cover up?

On the last entry, I flagged up when Mediamax disabled comments on its blog and deleted those that had been left and suggested Mediamax is not wanting its new customers to know about our concerns. Mediamax's John Hood responded on this point (though not the others raised since he last commented) to say:

"Huh?! Of late our whole blog is about problems with the system. We're hardly hiding anything. We've addressed every issue brought to us by users through support and posted it on the blog for all the world to see."

Okay, well what does it currently say on the Mediamax blog? "We installed new upload servers! We’re sure you’ve seen the improvement in upload processing because we’re seeing record upload levels...."

No suggestion of problems there.

A few more representative quotes from the blog:

24th July: "Uploading continues to work much better...."

21st July: "Thanks for your patience through our major migration to a new datacenter. The worst of it is behind us.... When you attempt to download these files, an “ooops and error occurred” message will appear. We are working to resolve this problem as soon as possible..... [no update that it is still not resolved nor indication of when it will be]"

11 July: "We’ve corrected most of the issues that resulted from the transition....."

2 July: "The transfer of our IT/DB operations to an outside vendor in Chicago is complete. Of course, when a move this large occurs there are going to be a few random issues to iron out. The known issues are listed below...." [Yup, this one was telling people of 'a few random issues', then nothing the till the 11 July post suggesting the problems are corrected].

This unofficial blog shows people are experiencing serious problems with both the system and customer support.

What is most disappointing is that despite John's 'Huh?!' comment, we are learning far more through Mediamax comments posted here than from the official blog.

For example, we have learned here from Mediamax:

28 July: "We don't allow comments on the blog because we don't the manpower to manage it and respond. We're a small start up of less than a dozen people....."

31 July: "There's an issue with some bit for bit copies, that compunds this issue, a new upload of an existing file won't work unless you change the content of the file itself. Just changing the file name won't work, you need to edit something like the metadata to change the checksum of the file. A new upload should then result in a working file. We are very sorry about this and are working to resolve it but please try the above workaround in the meantime."

And here's the most amazing one. An email from Mediamax posted as a comment on the last entry. This can also be seen at:
http://www.techcrunch.com/2007/07/26/streamload-upgrade-goes-very-wrong-some-users-revolt/

---Quote begins
Hi Michael,

We appreciate your willingness to hear Streamload’s feedback. Yes, Streamload has had a tough several weeks due to two serious issues on our part.

On June 15, Streamload had a major storage problem that has caused many of our customer files to become inaccessible. They are not gone forever, but it is taking a very long time to recover all the data because of the extraordinary amount of data stored. With our current recovery infrastructure, we can “only” recover about 10 terabytes per day, but we are adding more hardware to speed this process.

Combined with that, we had another project underway to move our entire infrastructure to a new datacenter with the goal of making it more efficient and cost effective for customers. The move did not go as planned since it took longer than we anticipated and there was much more customer impact than originally estimated.

With the new data center, MediaMax is backed with an all new technical infrastructure, which is fully redundant, has more capacity, and is faster. This new transition, as challenging as it has been, will ultimately allow us to provide better service and support more customers in the future.

On that note, your reader comments are right - a better job should have been done to communicate with affected customers. Though we may not have communicated quickly or with much detail, we are attempting to actively address concerns, and are exploring ways to address this issue and improve our customer service.

Having said that, three weeks ago I took over as CEO again and will be able to monitor the service and customer needs more closely and hopefully bring back the customer satisfaction that was built since I started the company 9 years ago.

-Steve Iverson
---quote ends


It seems to me that if Mediamax knows "we can “only” recover about 10 terabytes per day" it would be a calculation of a few seconds to calculate how long we have to wait till it is all recovered, working at the current rate.

Steve knows what is going on, but the answer I received from support on 30 July, posted here previously, said: "The fix is ongoing and in progress, but I don't have an ETA on it's completion. I'm sorry I don't have a better answer for you than that."

As Steve says: "On that note, your reader comments are right - a better job should have been done to communicate with affected customers."

Let me just repeat the comment from John Hood, Mediamax Director of Customer Support:

"Huh?! Of late our whole blog is about problems with the system. We're hardly hiding anything. We've addressed every issue brought to us by users through support and posted it on the blog for all the world to see."

Thursday, August 2, 2007

Mediamax says it's even better now!

According to the Mediamax blog, their system is now even better!

See:
http://mediamaxblog.blogspot.com/2007/08/site-update.html

---

We installed new upload servers! We’re sure you’ve seen the improvement in upload processing because we’re seeing record upload levels. Please let us know if you’re having issues with uploads at support@mediamax.com . Thanks for the files!

On a different front, you’ve probably seen that we’ve raised email sharing limits. We’re proud to say that our new db is ably handling the increased traffic. Please keep sharing!

As always, thanks for your continued support and your business!
----

At the time of writing comments are active on the Mediamax blog, so you may be able to give your feedback, but be quick.