Is there a design pattern for dealing with large datasets over the internet?

Question

I am looking for a design pattern that handles large data sets over the internet, and does periodic updating of these objects. I am developing an application that will display thousands of records in the UI at one time. Additionally, various properties on these objects are quite transient and need to be updated on the client to keep the user aware of the changing state of these records in the system. I have a few ideas how to approach this problem, but figured there might be a design pattern (or patterns) out there that handles this type of scenario.

Limitations:

1. The client-side for this is being written in Silverlight.
2. The objects themselves are not very big (about 15 value-type and string properties), but querying for all the data is expensive. The 15 or so properties contain data from various sources; no clever join statement or indexing is going to speed up the query. I am thinking of populating only a subset of the properties on initial load and then filling in the more expensive details as the user zooms in on a given grouping of objects. Think Google maps, but instead of streets and building it is showing the objects.
3. I will be able to limit the portion of the thousands of objects that are being updated. However, I will need the user to be able to "zoom out" of an context that allows granular updating to one that shows all the thousands of objects. I imagine that updating will be disabled again for objects when they leave a sufficient zoom context.

Ideas on how to tackle all or part of this problem? Like I mentioned I am considering a few ideas already, but nothing I have put together so far gives me a good feeling about the success of this project.

Edit:

I think the difficult parts really boil down to two things for which I may need two distinct patterns/practices/strategies:

1. Loading a large number of records over the internet (~5k).
2. Keeping a subset of these objects (~500) update-to-date over the internet.

There are several design patterns that can be used for everything else.

Edit 2:

Thanks for the links on various "push" implementation in Silverlight. I could swear sockets had been taken out of Silverlight but found a Silverlight 3 reference based on an answer below. This really wasn't a huge problem for me anyway and something I hadn't spent much time researching, so I am editing that out of the original text. Whether updates come down in polls or via push, the general design problems are still there. Its good to know I have options.

Edit 3: Follow-up on push technologies.

As I suspected the Silverlight WCF duplex implementation is comet-like push. This won't scale, and there are numerous articles about how it doesn't in the real world.

The sockets implementation in Silverlight is crippled in several ways. It looks like it is going to be useless in our scenario since the web server may sit behind any given client firewall that won't allow non-standard ports and Silverlight sockets won't connect on 80, 443, etc.

I am still thinking through using the WCFduplex approach in some limited way, but it looks like polling is going to be the answer.

Edit 4: Found a pattern to solve half my problem

I found this pattern (PDF) which illustrates the use of an iterator pattern to retrieve pages of data from the server and present them as a simple iterator. In .Net land I imagine this would be implemented as IEnumerable (samples code is in Java and Oracle SQL). Of particular interest to me was the asynchronous page prefetching, basically buffering the result set client-side. With 5k objects everything won't fit on the screen at once, so I can use a strategy of not getting everything at once yet hide that implementation detail from the UI. The core objects the app will be retrieving are in a database, then other look-ups are required to fully populate these objects. This methodology seems like a good approach to get some of the data out to the client fast.

I am now thinking of using this patter + some sort of proxy object pattern that listens for deltas to the result set and updates object accordingly. There are a couple of strategies one could take here. I could load all the data upfront, then send deltas of changes (which will probably need some additional code in the subsystems to provide notification of changes). This might be my first approach. I am still looking. Thanks for all the ideas so far.

Answer 1

The proxy design pattern is the pattern that will aid in transfering data from one point to another. The proxy design pattern will allow you to treat remote objects as if they were local.