Replica-BUG :
 
Client requires f+1 matching non-tentative replies to
send the next request. There can be two replicas which can execute
fast and allow clients to progress while having two replicas which
are slow and hence backlogged. However, the backlogged replicas send
commit messages for different protocol instance even though previous
messages may not be prepared at these replicas so that the fast replicas
have 2f+1 commit messages to ensure their progress. So the slow replicas
may end up having holes (that is they may not have
pre-prepare/prepare/commit) while the fast replicas continue to progress.
Meanwhile, fast replicas could execute checkpoint_interval number of
messages but cannot make these checkpoints stable until they receive
2f+1 matching checkpoints. Since the slow replicas do not commit messages
the checkpoints at fast replicas remain unstable. Fast replicas continue
to execute for max_out (=2*checkpoint_interval) messages and at this point
it takes the next checkpoint and blocks until all the previous checkpoints
are stable. Currently in BASE, a slow replica multicasts "status" message
that contains the information of all the missing messages to all the
replicas whenever the status timer expires. All the replicas respond to
this message by sending the missing messages that they may have. I have
observed that all that slow replicas may not receive all the messages at
one go to fill in all the holes. This problem worsens if it loses earlier
messages than later messages as it cannot execute the later messages
unless the previous messages are executed. When the holes are filled
the slow replicas execute in a burst and hence do not "recv" messages
during this period. When the load is high in the system, this results in
further loses. So it takes a while for the fast replicas can receive
enough checkpoint messages to progress. Hence, the performance tests
have huge variance based on  how the holes are created and filled. This
behavior is more prominent when running throughput tests with multiple
clients.
 
Fix :
 
Slow replica does not send a commit if prepared is not true for
all the previous messages and instead it sends a status message to fetch
the missing prepare messages from others replicas. Hence, when a fast
replica receive a commit message it can be sure that there are atleast
2f+1 replicas which have all the previous prepares, which will eventually
commit and execute the request and do not have to wait longer for making
checkpoints stable. By doing so we may restrict overlaped execution
of multiple instances but avoid the above behaviour where some of the
replicas fall behind badly and restricting the progress.
 
 
2> Client-Bug :
 
BASE has an optimization where client choses a server that sends the full
reply message and all others just send the digest. Client randomly choses
a server, when it is starts abd changes the replier only when it
retransmits the request for the second time. Also that all the replicas
send full reply for retransmitted requests increasing the load in the
network. In such case, a client can chose a slow replica as the replier
and client sends a request then it gets digests from the faster replicas
but fail to get the full reply. Then the client retransmits
the requests and all the fast replicas send the full reply before the
retransmit timer expires for the second time. Since the timer did not
expire for the second time, client does not change the replier. It keeps
repeating this and its performance is badly affected. This phenomenon
can be observed when you run multiple clients, where benchmarks running
on some clients finish cquickly and some others take longer time
(variance more than 50%).
 
Fix :
 
If reply digests arrive before the full reply we mark the current
replier as slow replica and for all the later messages we chose a replire
from the set of other replicas so that we change the replier even when
there are no multiple retranmissions and adapt to the responsiveness of
the replicas.

