Proposal: A Distributed Retrieval System with Provenance Propagation

Ian Jacobi

8 October 2009

Where Are We Going?

We are here.

Bibi / CC BY-NC-SA 2.0

shinyai / CC BY-NC 2.0

Mathieu Ramage / CC BY 2.0

How Do We Get There?

Localize!
Localize!
Localize!

Distributed

Localizing means there's a larger global scale.
Each node may only see a small portion of everything known.

While the global scope might hold a number of facts, each node may only be aware of a handful. Then, no one has to know everything.

Retrieval System

The problem is retrieving data to work on it.
Use "social networking" to solve the larger problem.

You learn facts from people who gathered them from other sources. If you like a fact, you can track down the source and learn more...

With Provenance

Difficult to carry around the entire provenance tree.
Solution:

Don't carry the provenance along with you.
Let the network do the storage for you.

Each datum in each node has a pointer to where it came from. By following the pointers back as far as possible, one can build the provenance trace.

Propagation

Data propagation model comes out naturally.
Building framework for generic propagator programming.

Each datum acts as a cell to which remote cells can subscribe. Each store also acts as a cell that lists the cells known. By subscribing in such a way, we can be notified when data changes.