Amazon Web Services Blog

…there would be no bugs.

I have been experimenting with S3 but find that there is a great deal of latency when requesting _known_ objects.

In my simplest test case, requesting a known object from a bucket which contains only two objects takes on average 180-220ms.  Note: this is not counting the time spent creating the request or sending the request but just the time that it takes for a PHP socket_read to return any data.  Requesting an object which does not exist takes on average 60-70ms.  Example code:

$obj = new s3();
$obj->connect();
$obj->requestObject($bucket, $key);    // generates a request and writes to the socket
$timer->start();
$data = $obj->readObject();    // reads response from the socket
error_log(sprintf("took %s sec", $timer->get_time()));

The readObject() function merely calls socket_read with a maximum number of bytes to return set to 4096.  The $key requested in the test case is not even this large.  Also, readObject() returns raw data and does not parse the response header.

In the following test case, the response time is rather quick leading me to believe that S3 just takes a while to actually locate a known object within S3:

$obj = new s3();
$obj->connect();
$obj->requestObject($bucket, $key);    // generates a request and writes to the socket
usleep(2000000);    // sleep for two seconds
$timer->start();
$data = $obj->readObject();    // reads response from the socket
error_log(sprintf("took %s sec", $timer->get_time()));

I am hoping to stream media from S3, but this delay is very noticeable when scanning through clips.  Will I be able to leverage S3 into my application?  Or will there always be a long delay while S3 locates my files on their storage network before sending me back some data?

Thanks in advance!

-Abe

I currently have a PHP application installed on an EC2 instance which acts as a proxy to S3 by buffering some data from S3 and then spitting it back to the client.  However, the cost for the initial PHP socket_read takes on average, 400-500ms.

So I ran a simple test script to see how long a single PHP socket_read would take from EC2 to S3 (without acting as a proxy) and it takes on average, 180-220ms, for the socket_read to complete (even if it only reads 1400 bytes or so of data).

I've also observed that if I request an object which does not exist in S3, the PHP socket_read completes rather quickly at around 25-60ms.

Has anyone attempted to use EC2 in this fashion where low latency is necessary for interactive applications?  Just wondering if this is a limitation of PHP's sockets implementation or a limitation of S3.  Thanks in advance!

I thought I'd post this here in case anyone runs across the same issue I did.


If you have a working Ubuntu 7.10 image, and you use ec2-bundle-vol to create a new image and then you can't log into the new image you're probably having the same problem I had, which is that the network wasn't working at all on the rebundled image. To confirm that you can look at the console output from the failed instance with "ec2gcons".


The issue was that the network device was trying to bind to the device with the same ethernet address that it had before. This is caused by /etc/udev/rules.d/70-persistent-net.rules. You need to make sure that file doesn't appear in the new image, the easiest thing is probably to just delete the file on the current image before re-bundling. I actually did it differently (I don't like my solution, I'd like to find a better one), I edited the AMI tools and added that file as a hard-coded exclusion.


Another way might be to remove /etc/udev/rules.d/75-persistent-net-generator.rules, which generates the 70-persistent-net.rules file.


If you think of a more elegant way let me know. Hope that helps!


Paul

So, I’m new here and used to do business through ebay. I got sick of the hidden fees, so checked out Amazon. Listed a few "test" items

We are using Amazon S3 as a CDN for our images, css, and JS files as part of the recommended steps in YSlow for Firebug. S3 is working well and we are able to provide caching support, but GZIP support does not seem to be available.

I have read some older posts about compressions from Amazon, but nothing recent. Is there a work around at the moment or does Amazon plan to release this feature soon?

Thanks,
Chris

Today's most popular headlines are Top 10 Free Video Rippers, Encoders, and Converters (39,112 views today), Make Your Linux Desktop Look Like a Mac (4,435) and Free Documents at DocStoc (2,987).

If you’ve been trying to download and install Quicksilver in order to get Leopard up to snuff, you’ve probably noticed that the Quicksilver site is down. In fact, it’s been down for quite a few days now. Fear not, Quicksilver lovers, as the application’s developer contacted us to let us know that the last version we posted about included a bug that caused some Quicksilver installations to ping for updates like crazy, which brought down the homepage. That, consequently, has meant no one has been able to download Quicksilver or any Quicksilver plugins for a few days. So, whether you want to install Quicksilver fresh or you’ve already got it installed, go grab the latest version from the link below (it’s a direct link) and help put out the fire for the folks who who so kindly bring us Quicksilver free of charge.

Did you know that you can jump straight to a Wikipedia search or start composing an email directly from Yahoo's search box? Weblog Digital Inspiration highlights five cool Yahoo search tricks—like the two mentioned above—that Google doesn't do.

For instance, type !wiki google in Yahoo! search box to search Google in Wikipedia.com. Other popular shortcuts are !ebay, !amazon, and !flickr. You can also execute these from the Firefox search box without changing the default search engine.

The post also highlights lyric search, ordering keywords in queries, and the linkdomain operator (which returns how many times one site links to another). It's not that these non-Google supported features will get many users to switch from Google to Yahoo, but some of the arguably worthwhile features might take you to Yahoo's doorstep on specific occasions—especially Yahoo's lyrics search.