Haizea

- Documented SlotTable and AvailabilityWindow.
- Minor changes in other files because of a few name changes in SlotTable and AvailabilityWindow.

Fixed bug where VM RRs in a cancelled lease were not being properly finalized (they were not being removed from the slot table, which caused trouble).

Also, Haizea was not sending shutdown commands to OpenNebula after a cancelled lease.

Add support for new ONE_AUTH in OpenNebula (instead of containing a username and password, the environment variable points to a file with this information)

Fixed: Estimating suspend/resume times with global filesystem still used old method to access requested memory

Fixed: Haizea would crash when fetching nodes for which OpenNebula had not yet collected monitoring information. This fix also has the pleasant side effect of allowing Haizea to detect new hosts added in OpenNebula without having to restart Haizea.

A few minor changes so Haizea will work with Python 2.4

Kludge so logging will work before Manager is fully initialized.

Updated configuration file parser; datafile is now in [accounting], not in [general]

Switched to a saner implementation for Singleton classes. Required multiple changes all around (which had the indirect side effect of making constructors less dependent on the get_config() function)

Finished utilization probes

Documented accounting module. Minor fixes to default probes.

Miscellaneous pylint-induced code cleanup.

Added script to automatically generate probe reference appendix.

Documented accounting probes.

Fixed whitespace that, while ok to Python, made epydoc barf.

Minor fixes for issues detected when running unit tests after last commits. Also removed several print statements that shouldn't be there.

Modified the accounting code so information is gathered using "probes" instead of inserting hardcoded statements throughout the Haizea code. Now, collecting additional information involves writing a pluggable probe, instructing the accounting module when the probe must be run (currently this can be at every timestep, when a lease is requested, or when a lease is done). Thus, gathering additional information doesn't require modifying the core Haizea code....

Created "pluggable" package, placed pluggable policies in it. This package will also contain pluggable accounting probes.

Made persistence configurable. Documented persistence code.

- Data persistence is implemented. It still needs some work (and has to be made configurable so it can be switched off during simulations), but it's working.
- Started to clean up the accounting code so it will collect data using "probes" instead of hard-coded statements.

Updated copyright notices.

Merged TP2.0/0.9 branch into trunk.

Minor documentation fix in haizea-convert-data

Removed analysis package; has been empty for a while and doesn't do anything

Lots of small fixes to issues uncovered during pre-release testing for TP1.3

Lots of small fixes to issues uncovered by the tests and by pylint

Minor fixes

Cleaned up exception handling in rm.py

Cleaned up code and added more comments, primarily in LeaseScheduler

Sanitized and improved exception handling

- Minor fixes to tests.
- Fixes of bugs uncovered by aforementioned tests.

Keep track of lease state using a state machine. Added more sanity checks that throw exceptions when lease is in an inconsistent state.

Miscellaneous code refactoring

OpenNebula 1.2 support

Fixed bug in per-experiment option of haizea-convert-data

Started refactoring scheduling code into more manageable modules + classes. Note that this leaves some parts of the scheduler broken for the time being.

Beginnings of Lease state machine

Made haizea-convert-data more useful

Fixed suspend/resume override

• Fork suspendresume rate into separate suspend and resume rates
• Take into account enactment overhead in shutdown (this includes adding a delay when sending commands to OpenNebula)

Minor changes in slottable implementation which should reduce running time of simulations.

Fixed bug where scheduler would crash in a pretty extreme corner case: a lease has to be scheduled in the future and the only reservation at the time happens to be a ShutdownResourceReservation. Since the scheduler is looking for VMRRs that are ending, it finds none.

Multiple small bug fixes

Apparently I got my branches messed up and I don't know how to do a proper merge yet... :-P

Attributes included in a multiconfig file are now passed along to the individual configuration files.

Previous revision was missing some code for this:

• Added an "override-memory" option to the config file to override the memory requested in a trace file. This is useful when running simulations where the memory is one of the variables. This should be replaced with a more general mechanism eventually.

• Multiple changes to slottable to improve performance of simulations
• Keep better track of resource utilization
• Added an "override-memory" option to the config file to override the memory requested in a trace file. This is useful when running simulations where the memory is one of the variables. This should be replaced with a more general mechanism eventually.

Explicitly schedule cold migrations

Take enactment overhead into account by spacing OpenNebula commands (avoiding OpenNebula from getting saturated)

- Modified suspend/resume scheduling so, when doing local exclusion, suspend/resumes happening at the same time are consolidated into a single RR.
- Included enactment overhead into suspend/resume scheduling
- Minor fixes

Added ShutdownResource Reservation to explicitly schedule shutdowns.

Changed VM RR data structure so that instead of including information on suspend/resume VMs, they include information on pre/post RRs (this will allow us to include other types of post RRs, such as shutdowns and live migrations)

Fixes some of the image transfer code, which was still using old nomenclature.

Missing configfile documentation

Minor fixes

Silly silly bug

Updated configuration file documentation

Small fixes, detected while testing for TP1.2

- Add dummy OpenNebula deployment enactment class
- Make RPC Server stoppable (and stop it when Haizea stops)

Adapted image transfer scheduler to new lease data structure topology

Adapted slideback algorithm to new lease data structure topology

Miscellaneous pending fixes

Adapted preemption code to new lease data structure topology. Preemption is working again.

Moved getNextLeasesScheduledInNodes to slottable

Added "dry run" to OpenNebula mode

Missing handlers in unmanaged deployment

No "dynamic loading" of enactment modules means no more silly directory structure

- Started removing lots of unnecessary dependencies, mostly references from objects to containing objects, which should be avoided. In particular, lots of objects don't have a "convenient" rm attribute pointing back to the Resource Manager. This attribute was only used to access the configuration file, the accounting object, and the clock. So, these three have been made "global" (a little globality is not bad, as long as it's controlled; in this case, the only way of accessing these objects is using the get_config, get_accounting, and get_clock global functions which, in turn, access the ResourceManager singleton)....

Misc. code cleaning

Add option to stop Haizea (in OpenNebula mode) when there are no leases left to process.

Add support for managing groups of OpenNebula VMs as a single lease.

Refactored new suspend/resume code

- Added support for globally and locally exclusive suspend/resume operations.
- Removed "what is the node doing?" accounting, since it assumed that only one VM is running per node (an assumption we're slowly getting rid of). There is currently no way of producing utilization statistics, and a new one (which accounts for multiple VMs, potentially heterogeneous ones, per physical node) will have to be added.

Miscellaneous code cleaning, most of it required by the previous two revisions.

Sanitizing of scheduling code:
- Moved more scheduling code out of the slottable and into the Scheduler class
- Factored out resumption/suspension scheduling code, to facilitate supporting more complex suspend/resume scenarios.
- Adapted code to new lease data structure topology...

Changed the topology of the lease data structure. Instead of containing a flat list of resource reservations, it now separates between deployment RRs and VM RRs, with the latter now being able to contain multiple suspend/resume RRs and migration RRs. This is necessary to accomodate complex suspend/resume scenarios (e.g., having multiple VMs on a physical node, saving memory files to a global filesystem, etc.)

Factored scheduling code out of SlotTable, and placed it in Scheduler

- Cleaned up image transfer + caching code
- Renamed "storage" enactment modules to (more appropriate) "deployment" enactment modules, and factored out some code from resourcepool and deployment modules there.

- Moved ResourcePool inside the Scheduler, instead of inside the ResourceManager
- Started cleaning up ResourcePool code (general clean up and factoring code out to the deployment modules)

NOTE: Only unmanaged deployment works right now. Image transfer scheduling is temporarily broken.

CLI documentation

Generate CLI documentation in LaTeX automatically

Added defaults module to contain default values, instead of hardcoding them throughout the code

Minor logging fixes

- Fixed get_queue RPC method
- Added get_hosts RPC method

Added XMLRPC marshalling method for hosts (for haizea-list-hosts)

- Made config options class-level variables instead of instance-level, so we can access the documentation without having to instantiate a configfile class.
- Added logfile option

Fixed logging

Uncommented logging

- Switched CLI implementation from functions to classes, mostly so they can be self-documenting
- Improved console pretty-printing
- Added haizea-show-queue and haizea-list-hosts

Decoupled lease data structures from scheduler

Changed logging code so there will be multiple loggers, accessible through logging.getLogger, instead of a single logger in the ResourceManager class (which required traversing the class hierarchy to get to that logger). Also made ResourceManager a singleton.

Miscellaneous small fixes

Adapt multiconfig to new configuration management. Save multiconfig attributes to accounting data object.

stats -> accounting

Changed name of stats.py to more descriptive "accounting"

Overhauled configuration file management. Lots of minor changes in most modules on account of this.

Added RPC/CLI for listing leases and cancelling leases

Logger can write to a file, instead of just to stderr

- Made clock configurable in simulation (i.e., user can choose the simulated clock that was being used so far, or a real clock where time is "real" and everything else is simulated)
- Moved wakeup-interval from opennebula section to scheduling section