March 10th, Jiandan

Option 1 does not work. Because the nextCausal will keep change.

Consider the following scenario:
-  A: 1, 2, 3
-  B: 1, 4

If A doesn't talked to B yet, then the causal linked list is A 1, A 2, A 3
If II.getNext(A 2), will return A3.

If A talked to B, the causal linked list will be A1 B1 A2 A3 B4.

If an InvalIterator returns A3 before A talked to B, then after A talked to B, 
getNext(A 3) will return B 4 and omitting B 1.



Two options of how to maintain the PerWriterLogIterator
(1) inside PerWriterLog

    Whenever the PerWriterLog applying new per-writer-log item,
    it checks each PerWriterLogIterator and sees if it needs to put new item
    into the PerWriterLogIterator.nextPointers so that the nextPointers
    reflect the new per-writer-log immediately.
    
    Cons: write/applyInval performance involves O(#iterator) updates
    pros: do it once, the iterator.getNext() only needs O(#iterator) search

    
    
(2) outside of PerWriterLog
    
    Whenever iterator.getNext() is called, it needs to check if there's
    anything new happened. O(#Iterator) search to identify new updates, 
    then another O(#Iterator) to find  the min.
    
    Cons: more overhead for each getNext(); even if no more new updates 
          happen.

    Pros: PerWriterLog is simpler.
          no affects to the local write/read performance    

The following pseudo code are for option (2).


Summary of Changes need:
=======================

(1) create PerWriterLogIterator class:
    An iterator that returns all invalidations in the per-writer-log
    in causal order by getNext();
    return null at the end;

    Difference between InvalIterator and PerWriterLogIterator:
      - performance: Besides cvv, it keeps the references of the
          next item to return for each writer so that the getNext will 
          be start from the last pointer instead of from the very begining.
	  
	  What's more, we precalculate the nextItem for each writer.
	  whenever iter.getNext() is called, it directly returns the min
	  of all current nextItem, and prefetch the next item to replace
	  the one to return.

      - semantics:
         . PerWriterLogIterator is block free.
	   It won't block. If no more item to return, it simply returns null.

         . PerWriterLogIterator won't accumulate any SingleWriterInv into
	   a general imprecise invalidation.
	   It only faithfully reports all the SingleWriterInv in the
	   current per-writer-log. Each SingleWriterInv is ordered by
	   it's start acceptStamp.


(2) create PerWriterLog class:
    move all the PerWriterLog code in UpdateLog to PerWriterLog

    --> clean up UpdateLog

(3) OutgoingConnectionEventMailbox:
    Coordinate the OutgoingConnectionWorker 
    between UpdateLog and OutgoingConnection. 

    Why?

    Issues with current design:
    
     OutgoingConnectionWorker is a single-thread for three tasks:
     1. deal with addSubscription/removeSubscription requests
     2. send catchup streams
     3. sending next new invalidation by calling UpdateLog.getNext()

    Currently, the possible blocking it has is:
     block at UpdateLog.getNext() due to no new updates which have a timeout
    
    During the blocking, the new coming addSub/removeSub requests can't be
    applied until it returns due to timeout if no new updates ever issued.


    Another issue with the worker is about Accumulating imprecise invalidation:
    currently, the accumulating is done in InvalIterator.
    Then it is possible that while InvalIterator is accumulating impreciseInv
    for a certain accumulatingTimeout period, there're new addSub/removeSub
    coming. Then what we will see is that the processing of addSub/removeSub 
    has a huge delay.


    Solution:
   
    add OutgoingConnectionEventMailbox to coordinate the worker and updateLog
    and outgoingConnection.

    OutgoingConnectionEventMailbox:
 
    1. block: 
       the worker only wait on OutgoingConnectionEventMailbox.getNextEvent() if 
       there's no pending requests or new updates.
 
    2. notify:
       OutgoingConnection will notify mailbox when there's new request
       UpdateLog will notify mailbox when there's new updates.

    move accumulating to Worker.

(4) IncomingConnection keeps a PerWiterLogPointer of the last item just applied
    to reduce the overhead of applying one item. right now. it's O(N).




PerWriterLogIterator:
===================


PerWriterLogPointers nextPointers;

CounterVV cvv; //summarize all sent invals
PerWriterLog log;

public 
PerWriterLogPointer(log){
  this.log = log;
  cvv = new CounterVV().makeAllVVNegatives();
  nextPointers = log.getNextPointers(cvv); //populate the next items to return
}


public 
PerWriterLogPointer(log, AcceptVV excludedStartVV){
  this.log = log;		
  cvv = new CounterVV(excludedStartVV);
  nextPointers = log.getNextPointers(cvv);//populate the next items to return
}

private InvalListItem
getNextPointerByNodeId(nodeId){
  return nextPointers.getInvalListItem(nodeId);
}


public GeneralInv
getNext(){
  //check if there's something new //search starting from the end
  newNextItems = log.getExcludedNextItems(cvv, nextPointers.keys());
  if(newNextItems != null){
  for each item in newNextItems{
    nextPointers.advance(item.getNodeId(), newNextItem);
  }

  nextItem = getMin();
  ret = nextItem.getInv();
  cvv.advance(ret.getEndVV());
  newNextItem = log.getNextItem(nextItem);
  nextPointers.advance(nextItem.getNodeId(), newNextItem);
}

private InvListItem getMin(){
  
}


/*
 * keep per-writer InvalListItem references for a number of nodes 
 */
PerWriterLogPointers:
====================
Hashmap<nodeId, InvListItem> pointers;

public PerWriterLogPointers(){
  pointers = new Hashmap();
}

public NodeId[]
getAllNodeIds(){

}

public InvalListItem
getInvalListItem(NodeId nid)
throws NullPointerException{
  return pointers.get(nid);
  
}

public InvalListItem
advance(NodeId newNID, InvalListItem newItem){
  assert newNID match newItem
  //if exists, assert newItem.as > oldItem.as
  //if not exist, put the new item.  
}


PerWriterLog:
============
HashMap<NodeId, SingleWriterLogUncommitted> perWriterLogs;

PerWriterLogPointers getNextPointers(AcceptVV startVV){
  ret = new PerWriterLogPointers();
  for each pwlog in perWriterLogs
    if pwLog.nodeId exists in startVV
      start = startVV[pwlog.nodeId]
    else
      start = -1;

    item = perWriterLog.getNextItemByStart(start);
    ret.advance(item);
  return ret;    
}

public InvalListItem[] 
getExcludedNextItems(AcceptVV sentVV, NodeId[] excludedNodes){
  NodeId[] diff = perWriterLogs.nodes() - excludedNodes;
  ret = new arrayList();
  for each node in diff
    ret.add(perWriterLogs[node].getNextItemByStart(sentVV[node]));
   

  return ret.toArray();

} 

public InvalListItem
getNextItem(InvalListItem currItem){
  assert currItem != null;
  return currItem.getNewer();
}

SingleWriterLogUncommitted:
===========================

public InvalListItem
getNextItemByStart(long start){
  //similar to getNextByStart
  //except that here we return the entire item
  // not just inv
}

InvalIterator::
==============

InvalIterator is very simple: just return the next non-gap-filling Object.
leave accumulation of imprecise invalidation to OutgoingConnection
Redesign for the getNext timeout and impreciseInvalidation timeout.

assumption: InvalListItem item's invaliadtion member can change, but
the invariant is that item.start will always be the same.

Replace CVV with PerWriterLogPointer: 
  InvalListItem currPos;// the current invalidate just returned.
			//    initially null,
			// it will never be a gap-filling item    
			//at the end, it remain the last one in the log.


return null or next causal;
Object getNext(){
     	
   newPos=updateLog.getNext(newPos, cvv);
   if (newPos == null){
     return nul;
   } else{
      newInv = newPos.getInv(); //note: newPos.inv might change, 
      //but it's ok because, the cvv only advances with the inv that it returns.
      //The invariant that the start won't change can guarantee that the invalidate returns will
      //be the next causal.
      cvv.advance(newInv.getEndVV());
      return newInv;
   }
} 



UpdateLog
=========

UpdateLog::per-writer-log head


UpdateLog::merge

UpdateLog.getNext(newPos, cvv){
  while((newPos!null) && (newPos.getNextCausal()).inv included in cvv)){
     newPos=newPos.getNextCausal();
  }
  return newPos.
}


SingleWriterLogUncommited:
=========================
/**
   change 1. add a new parameter currentPointer so that 
             it can search starting from the currentPointer.

          2. return an InvalListItem instead of a GeneralInval
*/

Another option: completely expose InvalListItem,
then InvalIterator can just call InvalListItem.getNextByStart(long start).

InvalListItem getNextByStart(InvalListItem, start){
  
}



OutgoingConnectionNextEventMailbox
==================================

queue, ii;

//called by OutgoingConnectionWorker
synchronized NextEvent getNext(){

  while (queue.isEmpty() && ii.hasNoNext()){
     wait();
  }
  if(!queue.isEmpty()){
    return queue.getNext();
  }
  if (ii.hasNewUpdate()){
    return ii.getNext();
  }

}


//called by OutgoingConnectionPool
synchronized void addNewRequest(newRequest){
   queue.add(newRequest);
   notifyAll();
}

//called by UpdateLog
//Note: don't put it under any lock.--> deadlock
synchronized void newUpdate(){
   notifyAll();
}



OutgoingConnection(){
  mailBox = new OutgoingConnectionMailbox();
  OutgoingConnectionMailBoxTable.add(mailbox);
}


OutgoingConnectionMailboxTable
==============================
   add(mailbox);
   getNewUpdates();//for all mailbox() call newUpdate();





Plan:

  incrementally plug-in the new code with the existing code.
  (1) implement SingleWriterLogIterator

  (2) replace log.getNext() in the original InvalIterator.getNext()
      with SingleWriterLogTierator.getNext()

  (3) Test the performance for the subscription set

  (4) change OutgoingConnection to use the new MailBox
      and use the new SingleWriterLogIterator

  (5) clean up old code completely including the UnbindMsg and DebargoMsg
