concurrent.framework.nodes package¶

Submodules¶

concurrent.framework.nodes.applicationnode module¶

Module containing a client node. Client nodes are the work requestor

class concurrent.framework.nodes.applicationnode.ApplicationNode(compmgr, init=None, cls=<class 'concurrent.framework.nodes.applicationnode.ApplicationNode'>)[source]¶

Bases: concurrent.core.components.component.Component, concurrent.framework.nodes.basenodes.Node

app_init()[source]¶

Initialize application just before running it

app_main()[source]¶

Launch a concurrent application

conditional_register_with_master()[source]¶

Try to register with master after an unexpected connection failure

generate_client_api()[source]¶

Generate the client API of our compute channel

get_master_address()[source]¶

Get the adress and port in (host,port) fashion

get_master_url()[source]¶

Get the URL where our master node is hosted

get_task_system()[source]¶

Called from the base class when we are connected to a MasterNode and we are able to send computation tasks over

has_master()[source]¶

Check if the node has a master or not. Master node has no master itself

master_disconnected(gracefully)[source]¶

Called when a master is disconnected (gracefully) or we had no response from the master itself (ungracefull)

master_url¶

This slave master node

push_task(task)[source]¶

Send a task to the computation framework

push_task_failed(result)[source]¶

We failed to add a Task to the computation framework

push_task_response(result)[source]¶

We just add a Task to the computation framework

push_tasks(tasks)[source]¶

Send a set of tasks to the computation framework

push_tasks_failed(result)[source]¶

We failed to add a set of Tasks to the computation framework

push_tasks_response(result)[source]¶

We just add a set of Tasks to the computation framework

push_tasksystem_failed(result)[source]¶

We failed to push a ITaskSystem on the computation framework!

push_tasksystem_response(result)[source]¶

We just added a ITaskSystem on the framwork. Check result for more info

register_client_failed(result)[source]¶

Called when we failed to register ouselfs to a master node. Raises an exception.

register_client_response(result)[source]¶

Called when we finsihed to register ouselfs to a master node. Raises an exception if the master rejected us.

register_with_master()[source]¶

The node will register itself with the expected master node

rpc_call_failed(proxy, method, reason)[source]¶

Called when an RPC call failed for an unexpected reason

rpc_call_success(proxy, method, result)[source]¶

Called when an RPC call succeded

send_heartbeat()[source]¶

Send heartbeat to master in case we have one

start_processing()[source]¶

Called when the app is not using a ITaskSystem and will instead just add tasks and will take care of the task flow itself

stop_app_node()[source]¶

task_finished(task, result, error)[source]¶

Called when a task has been done

unregister_from_master()[source]¶

The node will unregister itself with the expected master node

work_finished(result, task_system)[source]¶

Called when the work has been done, the results is what our ITaskSystem sent back to us. Check resukt for more info

concurrent.framework.nodes.basenodes module¶

Module containing our base node enteties

class concurrent.framework.nodes.basenodes.BaseNode[source]¶

Bases: object

Base node, all nodes will be atleast of this type. Responsible for hosting and exposing a simple API apart from listening on a TCP port for socket interactions.

aes_padding¶

Padding used for AES encryption

app_init()[source]¶

Initialize application just before running it

app_main()[source]¶

Launch a concurrent application

create_node_proxy(url)[source]¶

Create a new json proxy instance used by the node when acting as a client role

create_tcp_client_proxy(sock, request)[source]¶

Create a JSON TCP socket proxy instance to a client

create_tcp_client_proxy_zmq(context, identity)[source]¶

Create a JSON TCP socket proxy instance to a client

create_tcp_proxy(host, port)[source]¶

Create a JSON TCP socket proxy instance to a server

generate_api()[source]¶

get_stats()[source]¶

heartbeat_timer¶

Timer used to send periodically heartbeats to the master

index()[source]¶

main_loop()[source]¶

num_iterations¶

Number of iterations used in the key generation

on_update(delta_time)[source]¶

pickler¶

Pickler class used by the whole framework

ping()[source]¶

port¶

Port of the API interface with this node

proxy_api¶

API version used for any client JSON RPC calls

proxy_password¶

Password used when performing API client calls

proxy_username¶

Username used when performing API client calls

salt_size¶

Size of the salt used in the encryption process

secret¶

Default salt used on decrypting encrypting a pickle

shutdown_main_loop()[source]¶

stats_dump_timer¶

Timer used to dump stats into the log. -1 will never dump stats.

status()[source]¶

stop_api_thread()[source]¶

urls = ('/', 'index_get', '/ping/', 'ping_get', '/ping', 'pint_get', '/status/', 'status_get', '/status', 'status_get', '/stats/', 'stats_get', '/stats', 'stats_get', '/api/1/', 'APIHandlerV1', '/api/1', 'APIHandlerV1')¶

use_gzip¶

Check if we should gzip all interactions (recommended)

class concurrent.framework.nodes.basenodes.ComputeNode[source]¶

Bases: concurrent.framework.nodes.basenodes.Node

A compute node is a base node that does computation. It features a set of worker processes that except jobs from, usually a MasterNode.

generate_api()[source]¶

Create all rpc methods the node requires

get_num_workers()[source]¶

Return the number of workers the compute node has spawned

push_task(task)[source]¶

Push a task onto the computation framework

setup_compute_node()[source]¶

Launch the compute service from this node

stop_compute_node()[source]¶

task_finished(task, result, error)[source]¶

Called when a task has finished its computation, the result object contains the task, the result or an error and additional information

task_manager¶

Task manager used by this compute node

class concurrent.framework.nodes.basenodes.NodeType¶

Bases: object

Node types that life in the framework

client = 2¶

invalid = -1¶

master = 0¶

slave = 1¶

class concurrent.framework.nodes.basenodes.NodeState¶

Bases: object

Node state within the framework

active = 1¶

inactive = 2¶

inactive_forced = 3¶

invalid = -1¶

pending = 0¶

exception concurrent.framework.nodes.basenodes.FailedToRegisterWithMaster[source]¶

Bases: exceptions.Exception

Exception raised when a node failed to register itself with the master node onto the compute channel.

concurrent.framework.nodes.masternode module¶

Module containing the framework master node implementation

class concurrent.framework.nodes.masternode.MasterNode(compmgr, init=None, cls=<class 'concurrent.framework.nodes.masternode.MasterNode'>)[source]¶

Bases: concurrent.core.components.component.Component, concurrent.framework.nodes.basenodes.BaseNode

app_init()[source]¶

Initialize application just before running it

app_main()[source]¶

Launch a concurrent application

clean_node_map()[source]¶

Clean node map for any empty node values.

generate_api()[source]¶

Create all rpc methods the node requires

get_client(url)[source]¶

Get a node representation given an url

get_client_id(url)[source]¶

Return a client id given an url

get_client_id_no_lock(url)[source]¶

get_node(url)[source]¶

Get a node representation given an url

get_node_id(url)[source]¶

Return a node id given an url

get_node_id_no_lock(url)[source]¶

handle_echo(sock, address)[source]¶

has_master()[source]¶

Check if the node has a master or not. Master node has no master itself

heartbeat(node_id, node_type)[source]¶

We just received a nice beat from a node, update it’s last heartbeat timestamp to perevent timeouts

inactivity_time_multiplier¶

Inactivty multiplier multiplies the heartbeat time to ensure inactivity is always several heartbeats

is_standalone¶

Master node is also a slave and a standalone application

master_port¶

Port used by the master node for high-performance communication and dedicated persistent connections

notify_shutdown()[source]¶

Notify a global shutdown to all nodes

on_update(delta_time)[source]¶

push_task(request, task)[source]¶

We received a task from a client, add it to the system to be processed

push_tasksystem(request, tasksystem)[source]¶

We received a task system from a client. Get the first list of tasks and save out the system itself for later access

register_node(node_id, ip, port, data, node_type)[source]¶

Register a node within our node map

register_node_tcp(handler, request, node_id, node_type)[source]¶

Slave has just registered itself throug the compute channel

registry_cleanup_timer¶

Timer used to cleanup the node registry

registry_mirror_timer¶

Timer used to update node registry mirror

rpc_call_failed(proxy, method, reason)[source]¶

Called when an RPC call failed for an unexpected reason

rpc_call_success(proxy, method, result)[source]¶

Called when an RPC call succeded

set_registry_dirty()[source]¶

Set the registry dirty, this will force an update of the task scheduler

status()[source]¶

stop_master_thread()[source]¶

task_finished(task, result, error)[source]¶

Called when a task has finished its computation, the result object contains the task, the result or an error and additional information

task_scheduler¶

Task scheduler used by the master node

unregister_node(node_id, node_type)[source]¶

Unregister a node within our node map

update_inactive_nodes()[source]¶

Called when we check for inactive nodes, those that have not send any heartbeat for a while

update_registry_mirror()[source]¶

Update the registry mirror with a copy of the registry. Used to expose a copy dict to the public.

update_scheduler()[source]¶

Update task scheduler with the current list of slaves

concurrent.framework.nodes.slavenode module¶

Module containing a slave node. Slaves are computation nodes

class concurrent.framework.nodes.slavenode.SlaveNode(compmgr, init=None, cls=<class 'concurrent.framework.nodes.slavenode.SlaveNode'>)[source]¶

Bases: concurrent.core.components.component.Component, concurrent.framework.nodes.basenodes.ComputeNode

app_init()[source]¶

Initialize application just before running it

app_main()[source]¶

Launch a concurrent application

conditional_register_with_master()[source]¶

Try to register with master after an unexpected connection failure

generate_client_api()[source]¶

Generate the client API of our compute channel

get_master_address()[source]¶

Get the adress and port in (host,port) fashion

get_master_url()[source]¶

Get the URL where our master node is hosted

has_master()[source]¶

Check if the node has a master or not. Master node has no master itself

master_disconnected(gracefully)[source]¶

Called when a master is disconnected (gracefully) or we had no response from the master itself (ungracefull)

master_url¶

This slave master node

register_slave_failed(result)[source]¶

Called when we failed to register ouselfs to a master node. Raises an exception.

register_slave_response(result)[source]¶

Called when we finsihed to register ouselfs to a master node. Raises an exception if the master rejected us.

register_with_master()[source]¶

The node will register itself with the expected master node

rpc_call_failed(proxy, method, reason)[source]¶

Called when an RPC call failed for an unexpected reason

rpc_call_success(proxy, method, result)[source]¶

Called when an RPC call succeded

send_heartbeat()[source]¶

Send heartbeat to master in case we have one

stop_compute_node()[source]¶

task_finished(task, result, error)[source]¶

Called when a task has finished its computation, the result object contains the task, the result or an error and additional information

unregister_from_master()[source]¶

The node will unregister itself with the expected master node

Module contents¶