Lagniappe Programming Environment

Hard at Work

2009-02-19T00:15:00.003-06:00

Several directions are moving forward on the Lagniappe front.

Byzantine-fault tolerance is still our big target application. We have learned many interesting things about BFT, and how it differs from the traditional network applications we originally looked at.

We are currently in the process of writing BFT in Lagniappe, and in the next few weeks we should start to have the beginnings of the application take shape.

One interesting thing that Lagniappe allows us to do is look at BFT in terms of layered development and testing. The highly modular structure of Lagniappe makes this much easier. The small WiP presentation that I gave at OSDI talks about automatically applying fault-tolerance protocols to Lagniappe applications, and this layered breakdown of BFT is starting to clear up the details of how to make such automatic fault tolerance happen.

Stay tuned as we have exciting things that should be exiting the pipeline in the next few months!

OSDI WiP

2008-12-07T22:44:00.003-06:00

I've uploaded the WiP slides from OSDI.

The picture here has also been updated to properly reflect the structure of the lagniappe compiler in terms of the high-level transformations that occur to the models as we do our source to source translation.

There have been some updates in the code repository as we've added a replicated, ordered queue application (one of the components of a BFT RSM).

Thanks!

Moving forward

2008-06-24T23:37:00.004-05:00

I have my two initial benchmark applications coded up. They still need some testing, and I still need to finalize the workload generation, but having the apps down on the machine is nice.

I haven't touched the multi-machine code in a week. It just got to me after a while. The serialization aspect was killing me. I've decided to use the boost serialization library. At first I didn't think there was that much that I needed serialization for, but I eventually figured out that was not definitely not the case. Using the boost library will allow me to keep drastic changes between the trunk development of Lagniappe and the multi-machine branch, which will hopefully mean that the multi-machine support will be easier to merge back in when it's all said and done.

The work continues...

2008-06-11T10:12:00.003-05:00

While the OSDI deadline has come and gone, I am now making the final push towards defending and turning in my dissertation by the middle of August. Needless to say, this is the most important deadline I've ever run up against and it's not okay to pull the paper for this one.

I am working on two major tasks regarding the main system. The first is making Lagniappe function on multiple machines at the same time. If Lagniappe runs on multiple machines, the true heterogeneity of multi-* becomes an issue (as now we have a drastically different communication channel in the mix). The second task is combining operators based on their state preferences. I feel that we can truly answer the pipelined vs. parallel question based on the state access semantics of the operators in an application.

The third major issue is creating a suite of benchmark applications. The benchmarks has been a thorn in my side for a while, but I am confident we can make it all come together.

Busy, busy for OSDI

2008-04-29T22:51:00.001-05:00

Post OSDI I'll update the site on all of the new stuff that has been built into Lagniappe in the last month. It's really coming together, but there is still so much to do before the deadline.

Big changes

2008-04-09T14:15:00.005-05:00

So I've checked in code (in both the library and compiler) that automatically generates adaptation triggers based on the specifics of the application and system. I did a quick test and it does, in fact, generate different triggers for different hardware platforms, so that was a nice sanity check.

I've also now officially put all of the code under the GPL.

I'm going to spend the next few days on getting some of the paper up-to-date, and then I will start to tackle the smart mapping issue. With those two features on the books, the system will be pretty powerful. We will automatically generate policies to efficiently adapt resources to handle changes in workload based on your specific application and hardware platform. I think that is pretty powerful, but yes, I'm biased.

Update

2008-03-11T12:09:00.003-05:00

Things are starting to gel in my head and I'm starting a long section of implementation. Many features to get up and running and tested. Testing is something that I'm actually going to have to put some thought in. It is easy to test particular features in isolation, but the real trick in this work is watching the system deal with large-scale changes in incoming traffic. Generating the traffic that will properly trigger this activity and model real-life traces will be challenging.

The machines are here and installed in the machine room. They should be completely functional soon.

I will try to discuss the implementation as I go along.

So where are we?

2008-02-07T12:28:00.000-06:00

The Lagniappe project is embarking on very busy few months. With the OSDI deadline in early May, and my graduation in August, there is much work to be done.

There are now two other members on the Lagniappe team. Aruna and Rohan are helping me this semester. We are looking at many different avenues for showing Lagniappe's effectiveness.

We have some new hardware coming in that I am very excited about. We will have 32 cores to play with and these machines will allow us to really validate our ideas on a powerful, working system.

I am in the process of updating the Lagniappe web presence. I want to make sure that everything is correct, accurate, and up-to-date.

Wish us luck!

Proposal Time

2008-01-02T14:52:00.000-06:00

I am in the process of working on the presentation for my dissertation proposal. After that, I will get the updated goals and features up on the site as we start the final push to get Lagniappe full functional ahead of OSDI.

Busy Busy Busy

2007-10-05T10:44:00.000-05:00

We are deep in submission mode with the NSDI deadline on Tuesday at 7pm our time. Things are going okay, but it is going to be a tight and tough race to the finish. However, we have gotten some cool stuff done with the system in preparing for the submission, more details after we get it out. See you on the other side.

New versions

2007-09-14T13:26:00.000-05:00

New versions of both the Lagniappe compiler and library are up. I have added basic support for an architecture that supports adaptation. Also, application programmers now have access to the mutex implementation provided by the system programmer. We are starting the final several week push towards NSDI.

Packets

2007-06-26T13:56:00.000-05:00

The system can now actually process packets. Now the focus is on getting some basic monitoring and adaptation running. I hope to get some executable versions posted on the webpage soon. Of course, code can be checked out of the repository at anytime.

Very Close

2007-06-08T18:48:00.001-05:00

The system is very close to pushing packets... I am having to work through some interesting concurrent programming snags. However, I do think I finally figured out the right way to have the system programmer provide a mutual exclusion device to both me and the application programmer. Need to work out a few more logistical details, however.

The trick in all of this has been to maintain a painfully strict separation of application, system, and framework. The Lagniappe framework must maintain its platform neutrality. The same goes for the application. Things that one takes for granted like using locks get much harder when you don't know how those locks are going to be implemented, but you still need them.

ICCCN Paper Shipped

2007-05-31T14:49:00.000-05:00

So we sent the ICCCN camera ready off on Monday. I think it turned okay. I mean, there is always room for improvement, but I think the exercise overall was very productive. A copy of the paper can be found on the project's download space.

Now I get to get back to coding after a short hiatus to finish up the paper. I came to a realization on Sunday that the system has a really cool design. I mean, I know that sounds amazingly arrogant, but when I was explaining what was going on to Harrick and I saw him start to get excited as he started to understand the design, I realized, hey, I've actually done something pretty cool here.

So the image on the top of the page here isn't quite right. I haven't figured out the right image yet. I'm sure in the process of putting together the proposal talk I'll figure it out. The trick is to show graphically how the application is completely system independent (in other words, portable) and that the system code is obviously tied to the system. In the middle is the Lagniappe library that ties it all together to make a final executable that is tied to the system. Once we finally get some of the fancy stuff working (such as adaptation and monitoring) I think the power of the system will really start to come out.

Progress

2007-05-26T16:21:00.001-05:00

Whew... finally. I'm firing up threads, the communication channels are instantiated, the operators are created, as well as the application graph. I'm getting close to the point where I can start inserting packets into the system and watch it go up in fire. Hopefully, not too much fire.

I am hoping to get some initial proof of concept results by Monday at the latest. That is when the final version is due.

Big changes

2007-04-22T17:37:00.000-05:00

So I switched the code hosting from Sourceforge to Google Code. There interface is very clean and minimal. Sourceforge had gotten so cluttered and practically unusable.

The entire subversion tree was migrated over this afternoon, and I've already downloaded it into Eclipse. So check out the new Lagniappe page.

The block diagram at the top of the page also has changed drastically. We have completely redesigned the system, and I think it is much better now. I am currently trying to get a sort of proof of concept prototype built to validate our new ideas and to use in my proposal.

Now that most of the basic, foundation code is down I'm trying to decide exactly what features I want in the prototype. I want to get that feature set decided ASAP; too many possibles things to build leads to clutter in my brain. This thing doesn't need to do everything at the moment, I need to remember that so I can bang out what we need to show for the proposal.

Look for more updates soon as things are now rolling full speed.

Changes coming...

2007-04-18T14:38:00.000-05:00

New information coming. The project is really kicking off again. Look for major updates soon.

Updated.

2006-05-26T12:15:00.000-05:00

The llt and lee have been updated to work together. Get the files.

Trying to Adapt

2006-05-26T11:49:00.000-05:00

Movement. I've gotten the system back up to a working state after adding the initial code for adaptation. Right now, a very basic form of adaptation is done with packet level load balancing.

Not all of the design issues have been ironed out for doing efficient flow-level load balancing. There are also lots of little issues that start to arise once one starts reassigning resources. All of these will have to be ironed out before 0.5.0. However, getting packets flowing again was a nice accomplishment.

The big change is that the data processing threads now use pthread specific storage to keep track of exits and events. I had hacked up a custom version of this for the courier threads a while ago. I want to move the courier threads to the pthread system at some point as I believe it is superior to my hack.

The change to pthread specific also allowed me to remove a decent amount of overhead associated with each component execution. There had always been a wrapper function surrounding the data processing method provided by the user to catch the events and the exits. Now, because of thread specific storage, I do not need that wrapper. That should alleviate some function call and copy overhead.

Feel free to download the files for lee. The llt currently doesn't generate code that will work with it... but I should have that fixed this afternoon as the changes required are very minor.

Measuring work

2006-05-11T10:11:00.000-05:00

Right now I'm starting to put in the interfaces that will allow for "mutating channels," as we have called them in the poster proposal. Basically, when a PCon is enqueued at a processor, the amount of work that that component contributes will be recorded. Decisions are then made about resource allocation based on how much work processor can do (our old friend processor capacity).

The basic setup for this is pretty simple as far as runtime operation of lee is concerned. The trick is how to come up with these work numbers. We will need to profile the components at some point to try and get some type of work number.

I want to first get the mechanims for detecting overload, adapting resource allocation, and replicating components built first. Only then will I start to deal with the profiling problem.

Development plan updated.

2006-05-02T12:35:00.000-05:00

The development plan has been updated for milestone release v0.5.0. The major highlight is mutating channels for flow-based load balancing.

Released 0.4.0

2006-04-18T10:42:00.000-05:00

I uploaded the 0.4.0 milestone release yesterday. At the end of last week I tested the ability of the Lagniappe system to process packets off of the actual network. Since it seemed to be working, I was confident enough in the code to release the milestone and start to move on.

There is still much work to be done for the proposal. A little more urgent is the work to be done for the poster at PLDI. Getting the poster in will be good for me and the project as a whole. There are a few more features that need implementing (I hope to work on the development plan today and get those mapped out). Also, the poster has to be designed and some results generated. Lots of work. But fun work, things are progressing and I have the chance to tell more people about Lagniappe. Not to mention the chance to go to Ottawa, Canada and see the other great work being done in programming languages!

Non-code work

2006-02-28T10:10:00.000-06:00

Right now I'm focusing on my proposal, and thus I've had to stop the coding for a bit. I hope to start running tests in the near future so we can start to get a sense of the overhead of the system.

Device Discovery

2006-02-14T11:08:00.000-06:00

I really got device discovery implemented well now. The code in main.cc is a lot cleaner now as well because of this. The only thing I don't like is that I have to put a constant cap (at this point it is set to 10) of the devices to look for. The ideal way to do it is just have the system somehow return a set of usable devices... but I haven't figured a way to do that just yet. This way will work just fine for now.

I'm thinking I'm going to have to write/borrow a full-on packet access library. I need a way to construct sound packets for the major protocols. Pretty much, I want to be able to change a field in an TCP/IP packet and have an easy way for the checksums to be set and such. There has been lots of research into how to effectively do packet access. One of our graduates had a great way of taking a high-level specification of protocol header and generating efficient C++ that would allow you to access the header. For variable length fields (like options) it would give you an iterator that would automatically provide the right bounds on the number of fields.

I don't know if I need to get fancy... but I feel to effectively test the system I need to stop worrying that I've done the checksums properly and didn't munge up the packet headers. If I can push all that into a library it would be nice.

More progress

2006-02-08T18:30:00.000-06:00

I discovered (and implemented) today how to have the system determine the needed information about ethernet devices automatically. This way, when the system boots up it automatically determine the devices that are installed and instantiate them as objects visible to the programmer. This was a feature I wanted to have once real network access got going. I hope to flush the details out on this tomorrow and do some testing.