On the last entry, I flagged up when Mediamax disabled comments on its blog and deleted those that had been left and suggested Mediamax is not wanting its new customers to know about our concerns. Mediamax's John Hood responded on this point (though not the others raised since he last commented) to say:
"Huh?! Of late our whole blog is about problems with the system. We're hardly hiding anything. We've addressed every issue brought to us by users through support and posted it on the blog for all the world to see."
Okay, well what does it currently say on the Mediamax blog? "We installed new upload servers! We’re sure you’ve seen the improvement in upload processing because we’re seeing record upload levels...."
No suggestion of problems there.
A few more representative quotes from the blog:
24th July: "Uploading continues to work much better...."
21st July: "Thanks for your patience through our major migration to a new datacenter. The worst of it is behind us.... When you attempt to download these files, an “ooops and error occurred” message will appear. We are working to resolve this problem as soon as possible..... [no update that it is still not resolved nor indication of when it will be]"
11 July: "We’ve corrected most of the issues that resulted from the transition....."
2 July: "The transfer of our IT/DB operations to an outside vendor in Chicago is complete. Of course, when a move this large occurs there are going to be a few random issues to iron out. The known issues are listed below...." [Yup, this one was telling people of 'a few random issues', then nothing the till the 11 July post suggesting the problems are corrected].
This unofficial blog shows people are experiencing serious problems with both the system and customer support.
What is most disappointing is that despite John's 'Huh?!' comment, we are learning far more through Mediamax comments posted here than from the official blog.
For example, we have learned here from Mediamax:
28 July: "We don't allow comments on the blog because we don't the manpower to manage it and respond. We're a small start up of less than a dozen people....."
31 July: "There's an issue with some bit for bit copies, that compunds this issue, a new upload of an existing file won't work unless you change the content of the file itself. Just changing the file name won't work, you need to edit something like the metadata to change the checksum of the file. A new upload should then result in a working file. We are very sorry about this and are working to resolve it but please try the above workaround in the meantime."
And here's the most amazing one. An email from Mediamax posted as a comment on the last entry. This can also be seen at:
http://www.techcrunch.com/2007/07/26/streamload-upgrade-goes-very-wrong-some-users-revolt/---Quote begins
Hi Michael,
We appreciate your willingness to hear Streamload’s feedback. Yes, Streamload has had a tough several weeks due to two serious issues on our part.
On June 15, Streamload had a major storage problem that has caused many of our customer files to become inaccessible. They are not gone forever, but it is taking a very long time to recover all the data because of the extraordinary amount of data stored. With our current recovery infrastructure, we can “only” recover about 10 terabytes per day, but we are adding more hardware to speed this process.
Combined with that, we had another project underway to move our entire infrastructure to a new datacenter with the goal of making it more efficient and cost effective for customers. The move did not go as planned since it took longer than we anticipated and there was much more customer impact than originally estimated.
With the new data center, MediaMax is backed with an all new technical infrastructure, which is fully redundant, has more capacity, and is faster. This new transition, as challenging as it has been, will ultimately allow us to provide better service and support more customers in the future.
On that note, your reader comments are right - a better job should have been done to communicate with affected customers. Though we may not have communicated quickly or with much detail, we are attempting to actively address concerns, and are exploring ways to address this issue and improve our customer service.
Having said that, three weeks ago I took over as CEO again and will be able to monitor the service and customer needs more closely and hopefully bring back the customer satisfaction that was built since I started the company 9 years ago.
-Steve Iverson
---quote ends
It seems to me that if Mediamax knows "we can “only” recover about 10 terabytes per day" it would be a calculation of a few seconds to calculate how long we have to wait till it is all recovered, working at the current rate.
Steve knows what is going on, but the answer I received from support on 30 July, posted here previously, said: "The fix is ongoing and in progress, but I don't have an ETA on it's completion. I'm sorry I don't have a better answer for you than that."
As Steve says: "On that note, your reader comments are right - a better job should have been done to communicate with affected customers."
Let me just repeat the comment from John Hood, Mediamax Director of Customer Support:
"Huh?! Of late our whole blog is about problems with the system. We're hardly hiding anything. We've addressed every issue brought to us by users through support and posted it on the blog for all the world to see."
Allright, a try at a somewhat more indepth view of the demo I attended yesterday.
First of all one has to realise that V6 is still in development and that design clearly wasn't yet implemented ... i.o.w. it looked reasonably crappy, but I wasn't paying attention to how it looked, but as to how it functioned. Also I have to take into account that was just a demo on a limited database that can't compare with the MediaMax community as a whole so whatever I say is based on this demo. Final judgement will hopefully reflect the same feelings, but will not be made until I have experienced the final product myself.
During the demo Steve and John Hood said that they had taken a look at the wish list on this blog and that basically everything that was on there was or will be build into V6. I didn't see every wish being implemented during the demonstration, but these are the ones I can confirm and I have no doubt the others are or will be there as well. From my own list I saw item 2, 6, 8, 15-17 and 19-21. Afaik, there are no more demo's but if there is another one I'd love for people that attend to see if the others are already there as well.
Now for some highlights / points of interests:
- FILE RECOVERY PROCESS: basically this is finished and according to Steve they nearly got everything back that was accidentely deleted. Now I know Martin here has different figures which may have something to do with the entire system being unstable as I've seen files coming and going. The problem of file availability should be clear when the next step is taken ...
- CONVERSION: in order to go from V5 to V6 were going to have to endure another conversion. This is inevitable since they rewrote the entire code and atm this is expected to last 2-4 weeks when they flip the switch. A special note of interest is that MM will NOT convert links to files that are corrupt or no longer there. What this means is that every file you have in your V6 account is actually there and that there will be no more dead links or unavailable files like we had previously. Now in relation to file recovery process this means that basically until we're in V6 mode there will be no sure way to tell which files have been lost and which have been restored. I can only hope that MM is right about the number of files they have lost. Time will tell.
- SENDING: MM has reworked their sending system which may not to be everyones liking, but my guess it's a question of getting acustomed to it. Whether or not it solves one of their problems I'm not really sure. One of the problems MM is having is that their database is spiraling out of control with the number of lines it has to index/maintain.
Basically atm it works like this. If person A sends a folder to person B, then the database creates a copy of the the database lines for every file in that folder of person A and puts that into the account of person B. The problem seems to lie in the fact that people often send eachother a massive amount of files while only a small number is needed. Still the database has to make duplicate entries for each and everyone of those files.
In order to try and cut back on the number of lines their database has to create / maintain the new system will work like this (I hope I remember this right as I had a lot to keep up with attending the demo so correct me if I'm wrong): Person A sends person B a folder containing several files while person B is only interested in one. Instead of receiving that folder in the inbox like we were accustomed to, this folder now will be put on a share page from person A. If person B really wants it then person B has to go to the share page of person A, select the file and save it to his own account. Only when that has been done the database will create a new line for person B for that one file only and ignore the rest. To be honest I have my doubts if it will really work like this as what is to prevent someone saving the entire folder and then still only download the single file from his account?
- STORAGE: I'm not sure exactly how much is plan/wish or just mere thought, but it might be that MM is going to take a look at the amount of stuff people have stored. From what Steve said paid users will get a certain amount of space to store stuff on. People that have the same files will have the file counted against their storage capacity as file size divided by numer of people having the file (so a 100MB file present in 2 MM accounts, will count for 50MB against both accounts).
This will turn out to be a problem for large account holders that serve as an archive as they usually retain everything while the rest of the community benefits from that. You can rest assured I have mentioned this to Steve already. He said they basically don't want to change the way people use MM, but he said that some people are using MM in a way it was not intended storing massive logs on it and they they were looking at a way to clear of the dead weight on their servers. While I can understand their motive one of the edges MM has was their unlimited storage so cutting into that is not exactly the way to do it imho not to mention the fact that I (and others in my position) will be hurt by this. I do forego the probable grandfathering of old accounts, but it will mean that potential customers that are looking for a storage company probably will be put off. Also with the total amount of revenue they get they should be able to afford expanding their server capacity faster then we can fill it.
- UPLOADING: I'm told by someone who attended another demo then I did that FTP is postponed for now until at least the new service is operational, but that the old SL uploader will be back in a revamped fashion and that it will replace MM XL which is to be disbanded. Also I'm not quite sure how I should interpret this, but it was said that the processing stage of files in V6 will no longer be present. If you upload something it will appear straight away in your account.
- SHARING: basically the same thing what we now call hosting only in a working fashion and with a lot more options like auto expire, only visible for other (designated) MM account holders and a couple of more things I can't recall right now.
- FRIENDS / GROUPS: there was only minimum 'footage' of the Friends part. The only thing I recall offhand is that your friends will be incorporated in a quick list so if you have to send stuff regularly to the same account(s) you have short cuts for sending to them.
I hope I at least provided some glimpse as what V6 aspires to be and what is or may be to come. As I said earlier, I realise that this is only a demo and the real test will first be when we're allowed to alpha/beta test it, and after that when the MM community as a whole will be switched. IF, and I repeat IF, they manage to make the new system run as was presented at the demo then I think most of you will be happy to see the MM system back into action as it should have been from the beginning of V5. I've also made it clear that they will not get a 2nd chance. Failure this time is not an option. When they release V6 to the public they better be absolutely sure that it is indeed stable and reliable.