Technical Background of Digg's new Comment System

Posted on by
13

UPDATE: As you might have noticed, we’re having some technical issues rolling out the new comments. Please bare with us as we work out the kinks.

UPDATE: We reworked a few things and the comments are now live again.

Today Digg launched it’s redesign of the comment system, which was programmed by yours truly. Daniel has written up a detailed overview of the design decisions so if you’re interested in the design aspects you’ll definitely want to check that out.

There were a few fairly complex technical changes to the comment systems, which I’ll outline and then go into a little detail about.

  • We’ve been talking about moving towards a services oriented architecture on and off since I started in February. Steve had coded the API and the decision was made that the new comments system would use Digg’s public API.
  • Comment threads would be loaded dynamically using AJAX and JSON.
  • All commenting, editing, etc. would happen via AJAX.

The API ended up making the PHP code behind the scenes relatively painless. Each page loads using two small calls; one to figure out how many comments are on a story and another to fetch the first 50 or so comments (we use a fuzzy limit so it’ll just load all of them if there’s, say, 55 comments on a story). Both our local proxy and the comments code use the Services_Digg package we released via PEAR when the API launched. In fact, the entire permalink page is built using the API now, which is pretty neat.

By far the most complex portion of the comments system was how dynamic it was going to be. Threads would be zipping in and out, we’d be creating 90% of the HTML dynamically in the DOM from JSON, posting and editing over AJAX, etc. It was during design that Micah and I also plotted to remove script.aculo.us and replace it with the smaller jQuery library. The entire comment system is, in fact, a series of jQuery plugins.

Probably the coolest, technically speaking, portion of the new comments is the manner in which most of the page is created. No longer do we create static HTML in PHP and send you a huge HTML page. Instead we give you the basics and, via AJAX/JSON, we make requests to the API and dynamically create the DOM using the FlyDOM jQuery plugin. The FlyDOM JSON templates are a stroke of genius if you’re looking at loading JSON dynamically into the DOM. The advantage of this is that initial page loads are much snappier and you can load the threads you wish to read on demand.

I really picked up the whole dynamically created DOM ball and ran with if. If you notice, on the initial page load there aren’t any forms anywhere in the DOM. Those, also, are created dynamically on request. An interesting side effect to this is that there’s about 4x as much JavaScript code on the new comments than there is PHP.

The major technical and design changes of comments should lead to faster load times, less bandwidth being eaten up and, hopefully, a better user experience. I hope you enjoy them and, as always, welcome comments and input.

As Kevin would say, “Digg on!”

Google Developer Day 2007 – Advanced Topics on Google Data APIs

Posted on by
Reply
  • GData gives access to Apps, Base, Blogger, Calendar, Code Search, Notebook, Picasa Web Albums, and Spreadsheets. List is growing.
  • A protocol for reading and writing structured data over HTTP based on REST. A lot less complex than SOAP.
  • Atom Publishing Protocol uses HTTP GET to get feed data, HTTP POST to add a new entry, HTTP PUT to edit and HTTP DELETE to delete an entry. Interesting.
  • Authentication works through ClientLogin (highly trusted applications) and AuthSub (for less trusted web applications).
  • AuthSub is works like Flickr’s authentication model as well.

Google Developer Day 2007 – New Features in the Google Maps API

Posted on by
Reply
  • MarkerManager manages which makers to add depending on how large the viewport is. All markers are in memory, but that’s fine since markers are really small. Makes sure you don’t have 20,000 markers on a map of the whole US.
  • You can use map.addOverlay with GGeoXML, which uses KML, to overlay geo data onto maps (and Google Earth). They also support GeoRSS. He’s showing 6 lines of code overlaying interesting airplanes throughout the US.
  • They’re showing off Tile Layer Overlay, which allows you to overlay tiles on maps. Tiles are based on X, Y and Z (Z = Zoom). The example is of Stanford’s college campus map overlayed on top of the standard Google Map. Hot. You can set transparency on tiles too.
  • They use the Mercator Projection (must investigate this).
  • No way to easily say “I don’t have data outside of this boundary box”.
  • Driving directions just launched in the Google Maps API. Four lines of code to create a map with drections. Insanity. Also, allows you to load “Way Points” (ie. directions from Foo to Bar and then Bar to Baz) along with locales.
  • 10,000 requests per key per day.
  • They’re now embedding sponsored links directly into the maps. You’ll be able to enable Google Adsense on your maps. This is an opt-in feature. This would indicate that advertisers can now geo tag their advertisements. Advertisers can be blacklisted, ads are shown based on viewport and HTML content around the map.
  • Plans are in motion to base traditional AdSense ads shown outside of the map on the actual map dat.
  • All of the AdSense stuff above should launch sometime in June.

Google Developer Day 2007 – Enterprise Search APIs: Making Information Accessible at Work

Posted on by
Reply
  • The usual presenter technical difficulties to start things off.
  • An entire group devoted to enterprise was created in 2002 with over 7,000 customer of all shapes and sizes.
  • Search (Google Search Appliance), Geospatial (Maps and Earth), Collaboration (Google Appes).
  • You can post content to the Google Search Appliance (similar to Google Site Maps for the GSA).
  • GSA Feeds Protocol allows you to add meta data to each record, which can then be used to filter search results.
  • You can feed the content of the document into the index as well, which allows you to control the actual content that’s indexed. Interestingly, you can post binary data formats as well.
  • You can group content into “collections” which are based on URL patterns.
  • XSL stylesheets for search results (already knew this. You can also query via the GSA’s XML interface).
  • Based on URL patterns you can mark some content secure, and others unsecure. Along with an access parameter to restrict search results. Additionally, the search appliance can ping a callback URL via HTTP GET with the user’s cookies and show results based on response (401 = unauthorized, 200 = authorized).
  • The GSA’s now allow you to install “OneBox”‘s (e.g. weather, FedEx, etc. data from Google) that shows dynamic information in search results. Triggered by a keyword and then grabs information from a provider URL based on that query. OneBox’s trigger an HTTP request to the provider URL, which returns XML that is then translated by your OneBox’s results XSL template. Insanely cool.
  • GSA’s can do up to 30,000,000 documents and beyond with custom setups.

Google Developer Day 2007 – YouTube API

Posted on by
Reply
  • YouTube API philosophy is to get YouTube everywhere, give flexibility, maintain some consistency of overall YouTube user experience and, respect developers and content creators.
  • Just showed a new “beta” video player that has screenshots scrolling along the bottom of featured videos.
  • http://youtube.com/rssls for more information on their RSS feeds.
  • Offer REST and XML-RPC API’s. XML-RPC returns REST’s XML response escaped as an XML-RPC data structure. Seems a bit hacky to me.
  • Standard feed responses include thumbnail images.
  • Working on a new set of API’s that conform to the GData API standards. They’re investigating open/upload/write functionality. New search calls, filtertering, etc.
  • They’re working on adding ability to show a specific frame number as opposed to whatever thumbnail they give you by default. No release date.
  • They’re working on adding more programs where users can create businesses around their content. Those who use an app to manage Google My Business listings know what I mean by good content and how to use it to the full.
  • Also working on allowing users to do seeking, etc. from the API (ie. start the video from a specific start point). No planned support for allowing JavaScript to control the player.
  • For the foreseeable future they do not plan on allowing access to the raw videos (e.g. usurp the embedded player).
  • Lots of plans for extensive statistics for YouTube video viewing (e.g. how much of a video was watched, who’s watching, etc.).
  • They are working on an upload API, which is currently focused on device manufacturers and not necessarily 100% open to all (what about GData?).
  • Search only allows searching meta data, but implicit data (e.g. voice recognition) isn’t searchable. It doesn’t sound like they’re working on it, but Google being Google you should expect this (didn’t they just launch face recognition for image search?).
  • Geo tagging and deep geo tagging is on their roadmap. They’re going to be adding steps to the upload process that should allow better information. Once there’s enough of that data they’ll be exposing it.
  • Doesn’t exactly sound like they’re working on editing, but that they’re interested in it.
  • Google Analytics? Haven’t done any work to allow such things, but the hope is that stats will be available from them and exposed publicly.

Introducing correlate.us

Posted on by
2

As some of you know I’ve been working on a little side project and I’d like to officially announce that correlate.us is open for business. Whatever that means.

I’ve been kicking this idea around for quite some time. The basic idea is that you authenticate your various Web 2.0 accounts and then correlate.us goes out and aggregates that data, groups it by tags and makes it generally more browseable for everyone involved.

The best part about it is that you can add friends. Once you’ve added a few friends you can view your friends’ activity online in an aggregated view as well. Don’t forget to add correlate.us to your facebook profile.

Intelligent image thumbnails

Posted on by
Reply

A recent project I’ve been working on required 4:3 thumbnails of all images no matter the original image’s orientation. This required me to do a little math and cropping before actually making the thumbnail. I use the package Image_Transform to figure out which orientation the image is and then crop it appropriately.


$o = Image_Transform::factory('IM');
if (PEAR::isError($o)) {
    return $o;
}

$result = $o->load($ds1);
if (PEAR::isError($result)) {
    return $result;
}

$w = $o->getImageWidth();
$h = $o->getImageHeight();

if ($h > $w) {
    $newWidth = $w;
    $newHeight = ($newWidth * .75);
    $newY = ($h * .15);
    $newX = 0;
} else {
    $newWidth = ($w / 2);
    $newHeight = ($newWidth * .75);
    $newY = ($h / 2) - ($newHeight / 2);
    $newX = ($x / 2) + ($newWidth / 2);
}

$o->crop($newWidth, $newHeight, $newX, $newY);
$o->save($ds2);

The first check checks to see if the image is in landscape or portrait (in portrait the height will be greater than the width). With portrait’s I use the width, multiple that by 0.75 to get my 4:3 ratio and finish by going 15% down from the top of the portrait (assuming that you’re focusing your portrait in the upper portion of the image). With landscape images I simply go outwards from the center of the image (again, assuming you’re focusing your landscape in the center of the image).

The Rick Mathes chain letter

Posted on by
Reply

More fan mail from my family and friends who keep sending me stuff that you’d think they wouldn’t forward their liberal, tree hugging, hippy relative who lives in the land of “The Gays” and doesn’t even own a car. Can you imagine? No car?! I mean, he rides a bike to work! The most recent is a letter written by Rick Mathes about his run-in with a Muslim imam at a prison workshop. The letter asserts:

  1. The Muslim religion is the fastest growing religion per capita in the United States, especially amongst minorities.
  2. That Muslims are assured a place in heaven by killing an infidel.

I’m really not even sure where to start, but I’ll give it a shot. So here are a few counterpoints to this letter. Feel free to pass them on to your own friends and family if and when you get this chain letter.

  1. Actually, the fastest growing religious identity in the United States is “none”. In fact, the number of people describing themselves as having no religious identity has doubled since 1990.
  2. The definition of infidel (a term originally coined by Christians to describe Muslims during the Crusades) is “a non-believer”. Most Muslims generally accept that Christians and Jews believe in God, which makes sense when you take into account that Jews, Christians and Muslims all believe in the same God. The major sticking point is exactly who the messiah is; Jews don’t believe he/she has come yet, Christians believe it was Jesus and Muslims believe it was Muhammad. It should also be pointed out that the Old Testament is the Jewish Torah and that Muslims believe both the Torah and Bible to be the gospel of God.
  3. Don’t miss the Snopes.com page, which includes a reversed version of the letter and refutes the claim the minister was talking to an actual imam.

If you're so religous why did you choose that job?

Posted on by
Reply

Today the Metropolitan Airports Commission voted 11-0 in favor of penalties for cab drivers who refuse to transport drivers who are carrying alcohol and other legal merchandise that might be against Muslim religion. Now, if we could get the American Pharmaceutical Association to do the same for those religious zealots that refuse to hand out birth control and the day after pill we’ll be all set.

Seriously, if you’re such a religious zealot why on God’s/Allah’s green earth are you performing said job? I know that I, personally, would never take a job that might require me to ignore my own moral compass, so why did you?

What’s next? Jewish waiters refusing to serve me bacon?

I'm going to Thailand

Posted on by
2

Well, it’s all set in stone. I bought a ticket this morning whose final destination is Bangkok, Thailand. I’m planning on spending a few days in Bangkok and then taking a flight down to the Surat Thani Province for the Koh Phangan full moon beach party.

If anyone has any input on what gear I should take with me, any sites I should definitely see, etc. I’d love to hear about it. Post it in the comments for all to share.