Base interface for async tasks classes
Bases: concurrent.core.components.component.Interface
Interface used to define what a task manager is for us
Bases: object
A task system is an implemented system that provides a scheduler with tasks to be executed
Once the system stated that it has finsihed the MasterNode will request the required results that are to be send to the originator. Returns a tuple like (result, Error)
Bases: concurrent.core.components.component.Interface
Interface used by our distributed task scheduler. A scheduler receives an implemented system that will be executed on the distributed system through pickleing Python instances.
Bases: concurrent.core.components.component.Interface
Implements a schedule strategy to select the next valid slave that should process a given task
Base class for any of our async tasks that a job is made of.
Bases: object
A simple tasks that just executes a function in a fire and forget way
Called once a task has been performed and its results are about to be sent back. This is used to optimize our network and to cleanup the tasks input data
Bases: concurrent.core.components.component.Component
Number of worker processed to be created, -1 will spawn as much as physical cores.
Module containing various base thread classes and threading helpers
Bases: threading.Thread
A thread class that supports raising exception in the thread from another thread. Based on the class from Bluebird75 [http://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python]
Raises the given exception type in the context of this thread.
If the thread is busy in a system call (time.sleep(), socket.accept(), ...), the exception is simply ignored.
If you are sure that your exception should terminate the thread, one way to ensure that it works is:
t = ThreadWithExc( ... ) ... t.raiseExc( SomeException ) while t.isAlive():
time.sleep( 0.1 ) t.raiseExc( SomeException )
If the exception is to be caught by the thread, you need a way to check that your thread has caught it.
CAREFUL : this function is executed in the context of the caller thread, to raise an excpetion in the context of the thread represented by this instance.
Bases: exceptions.Exception
Error used when an interruptible thread has been killed
Bases: object
Read-Write lock class. A read-write lock differs from a standard threading.RLock() by allowing multiple threads to simultaneously hold a read lock, while allowing only a single thread to hold a write lock at the same point of time.
When a read lock is requested while a write lock is held, the reader is blocked; when a write lock is requested while another write lock is held or there are read locks, the writer is blocked.
Writers are always preferred by this implementation: if there are blocked threads waiting for a write lock, current readers may request more read locks (which they eventually should free, as they starve the waiting writers otherwise), but a new thread requesting a read lock will not be granted one, and block. This might mean starvation for readers if two writer threads interweave their calls to acquireWrite() without leaving a window only for readers.
In case a current reader requests a write lock, this can and will be satisfied without giving up the read locks first, but, only one thread may perform this kind of lock upgrade, as a deadlock would otherwise occur. After the write lock has been granted, the thread will hold a full write lock, and not be downgraded after the upgrading call to acquireWrite() has been match by a corresponding release().
Acquire a read lock for the current thread, waiting at most timeout seconds or doing a non-blocking check in case timeout is <= 0.
In case timeout is None, the call to acquireRead blocks until the lock request can be serviced.
In case the timeout expires before the lock could be serviced, a RuntimeError is thrown.
Acquire a write lock for the current thread, waiting at most timeout seconds or doing a non-blocking check in case timeout is <= 0.
In case the write lock cannot be serviced due to the deadlock condition mentioned above, a ValueError is raised.
In case timeout is None, the call to acquireWrite blocks until the lock request can be serviced.
In case the timeout expires before the lock could be serviced, a RuntimeError is thrown.
Bases: object
A simple class used to hold a cache of locks. Locks are being accessed as they where members of the class itself.
Create several work requests for same callable with different arguments.
Convenience function for creating several work requests for the same callable where each invocation of the callable receives different values for its arguments.
args_list contains the parameters for each invocation of callable. Each item in args_list should be either a 2-item tuple of the list of positional arguments and a dictionary of keyword arguments or a single, non-tuple argument.
See docstring for WorkRequest for info on callback and exc_callback.
Bases: exceptions.Exception
All work requests have been processed.
Bases: exceptions.Exception
No worker threads available to process remaining requests.
A thread pool, distributing work requests and collecting results.
See the module docstring for more information.
Add num_workers worker threads to the pool.
poll_timout sets the interval in seconds (int or float) for how ofte threads should check whether they are dismissed, while waiting for requests.
Tell num_workers worker threads to quit after their current task.
Perform Thread.join() on all worker threads that have been dismissed.
A request to execute a callable for putting in the request queue later.
See the module function makeRequests for the common case where you want to build several WorkRequest objects for the same callable but with different arguments for each call.
Bases: threading.Thread
Background thread connected to the requests/results queues.
A worker thread sits in the background and picks up work requests from one queue and puts the results in another until it is dismissed.