mirror
/
moonstream
kopia lustrzana https://github.com/bugout-dev/moonstream
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność

Revert "Supports tags filter for nodes"

pull/749/head
Sergei Sumarokov 2023-01-25 16:37:57 +03:00 zatwierdzone przez GitHub
rodzic 2659ad91b9
commit 3f814c779e
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
10 zmienionych plików z 187 dodań i 290 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

65
nodes/node_balancer/README.md
Wyświetl plik

@ -1,49 +1,17 @@
# Node Balancer application # Node Balancer application
## Installation and configuration # Installation
- Prepare environment variables, according with `sample.env`. - Prepare environment variables
- Build application - Build application
```bash ```bash
go build -o nodebalancer . go build -o nodebalancer .
``` ```
- Generate configuration # Work with nodebalancer
```bash ## add-access
nodebalancer generate-config
```
- Modify configuration. Tags should NOT repeat blockchain, as it is specified in `blockchain` key. Example of configuration:
```bash
[
{
"blockchain": "ethereum",
"endpoint": "http://127.0.0.1:8545",
"tags": ["local"]
},
{
"blockchain": "ethereum",
"endpoint": "http://127.0.0.1:9585",
"tags": ["local"]
},
{
"blockchain": "ethereum",
"endpoint": "https://cool-name.quiknode.pro/y0urn0de1den1f1cat0r/",
"tags": ["external"]
}
]
```
So if with request will be specified tag `local` will be returned node with corresponding tag.
## Work with nodebalancer
**IMPORTANT** Do not use flag `-debug` in production.
### add-access
Add new access for user: Add new access for user:
@ -57,7 +25,7 @@ nodebalancer add-access \
--blockchain--access true --blockchain--access true
``` ```
### delete-access ## delete-access
Delete user access: Delete user access:
@ -69,7 +37,7 @@ nodebalancer delete-access \
If `access-id` not specified, all user accesses will be deleted. If `access-id` not specified, all user accesses will be deleted.
### users ## users
```bash ```bash
nodebalancer users | jq . nodebalancer users | jq .
@ -99,7 +67,7 @@ This command will return a list of bugout resources of registered users to acces
`extended_methods` - boolean which allow you to call not whitelisted method to blockchain node, by default for new user this is equal to `false` `extended_methods` - boolean which allow you to call not whitelisted method to blockchain node, by default for new user this is equal to `false`
### server ## server
```bash ```bash
nodebalancer server -host 0.0.0.0 -port 8544 -healthcheck nodebalancer server -host 0.0.0.0 -port 8544 -healthcheck
@ -108,17 +76,17 @@ nodebalancer server -host 0.0.0.0 -port 8544 -healthcheck
Flag `--healthcheck` will execute background process to ping-pong available nodes to keep their status and current block number. Flag `--healthcheck` will execute background process to ping-pong available nodes to keep their status and current block number.
Flag `--debug` will extend output of each request to server and healthchecks summary. Flag `--debug` will extend output of each request to server and healthchecks summary.
## Work with node # Work with node
Common request to fetch block number Common request to fetch block number
```bash ```bash
curl --request POST 'http://127.0.0.1:8544/nb/ethereum/jsonrpc?access_id=<access_id>&data_source=<blockchain/database>' \ curl --request GET 'http://127.0.0.1:8544/nb/ethereum/jsonrpc?access_id=<access_id>&data_source=<blockchain/database>' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
--data-raw '{ --data-raw '{
"jsonrpc":"2.0", "jsonrpc":"2.0",
"method":"eth_getBlockByNumber", "method":"eth_getBlockByNumber",
"params":["latest", false], "params":["0xb71b64", false],
"id":1 "id":1
}' }'
``` ```
@ -129,16 +97,3 @@ For Web3 providers `access_id` and `data_source` could be specified in headers
--header 'x-node-balancer-data-source: <blockchain/database>' --header 'x-node-balancer-data-source: <blockchain/database>'
--header 'x-node-balancer-access-id: <access_id>' --header 'x-node-balancer-access-id: <access_id>'
``` ```
Same request to fetch specific nodes using tags
```bash
curl --request POST 'http://127.0.0.1:8544/nb/ethereum/jsonrpc?access_id=<access_id>&data_source=<blockchain/database>&tag=<specific_tag_1>&tag=<specific_tag_2>' \
--header 'Content-Type: application/json' \
--data-raw '{
"jsonrpc":"2.0",
"method":"eth_getBlockByNumber",
"params":["latest", false],
"id":1
}'
```

213
nodes/node_balancer/cmd/nodebalancer/balancer.go
Wyświetl plik

@ -1,5 +1,5 @@
/* /*
Load balancer logic. Load balancer, based on https://github.com/kasvith/simplelb/
*/ */
package main package main
@ -19,9 +19,10 @@ import (
// Main variable of pool of blockchains which contains pool of nodes // Main variable of pool of blockchains which contains pool of nodes
// for each blockchain we work during session. // for each blockchain we work during session.
var blockchainPools map[string]*NodePool var blockchainPool BlockchainPool
// Node structure with // Node structure with
// StatusURL for status server at node endpoint
// Endpoint for geth/bor/etc node http.server endpoint // Endpoint for geth/bor/etc node http.server endpoint
type Node struct { type Node struct {
Endpoint *url.URL Endpoint *url.URL
@ -35,16 +36,16 @@ type Node struct {
GethReverseProxy *httputil.ReverseProxy GethReverseProxy *httputil.ReverseProxy
} }
type TopNodeBlock struct { type NodePool struct {
Block uint64 Blockchain string
Node *Node Nodes []*Node
// Counter to observe all nodes
Current uint64
} }
type NodePool struct { type BlockchainPool struct {
NodesMap map[string][]*Node Blockchains []*NodePool
NodesSet []*Node
TopNode TopNodeBlock
} }
// Node status response struct for HealthCheck // Node status response struct for HealthCheck
@ -57,25 +58,24 @@ type NodeStatusResponse struct {
} }
// AddNode to the nodes pool // AddNode to the nodes pool
func AddNode(blockchain string, tags []string, node *Node) { func (bpool *BlockchainPool) AddNode(node *Node, blockchain string) {
if blockchainPools == nil { var nodePool *NodePool
blockchainPools = make(map[string]*NodePool) for _, b := range bpool.Blockchains {
} if b.Blockchain == blockchain {
if blockchainPools[blockchain] == nil { nodePool = b
blockchainPools[blockchain] = &NodePool{} }
}
if blockchainPools[blockchain].NodesMap == nil {
blockchainPools[blockchain].NodesMap = make(map[string][]*Node)
}
blockchainPools[blockchain].NodesSet = append(blockchainPools[blockchain].NodesSet, node)
for _, tag := range tags {
blockchainPools[blockchain].NodesMap[tag] = append(
blockchainPools[blockchain].NodesMap[tag],
node,
)
} }
// Check if blockchain not yet in pool
if nodePool == nil {
nodePool = &NodePool{
Blockchain: blockchain,
}
nodePool.Nodes = append(nodePool.Nodes, node)
bpool.Blockchains = append(bpool.Blockchains, nodePool)
} else {
nodePool.Nodes = append(nodePool.Nodes, node)
}
} }
// SetAlive with mutex for exact node // SetAlive with mutex for exact node
@ -105,76 +105,71 @@ func (node *Node) UpdateNodeState(currentBlock uint64, alive bool) (callCounter
return callCounter return callCounter
} }
// FilterTagsNodes returns nodes with provided tags // IncreaseCallCounter increased to 1 each time node called
func (npool *NodePool) FilterTagsNodes(tags []string) ([]*Node, TopNodeBlock) { func (node *Node) IncreaseCallCounter() {
nodesMap := npool.NodesMap node.mux.Lock()
nodesSet := npool.NodesSet if node.CallCounter >= NB_MAX_COUNTER_NUMBER {
log.Printf("Number of calls for node %s reached %d limit, reset the counter.", node.Endpoint, NB_MAX_COUNTER_NUMBER)
tagSet := make(map[string]map[*Node]bool) node.CallCounter = uint64(0)
} else {
for tag, nodes := range nodesMap { node.CallCounter++
if tagSet[tag] == nil {
tagSet[tag] = make(map[*Node]bool)
}
for _, node := range nodes {
tagSet[tag][node] = true
}
} }
node.mux.Unlock()
topNode := TopNodeBlock{}
var filteredNodes []*Node
for _, node := range nodesSet {
accept := true
for _, tag := range tags {
if tagSet[tag][node] != true {
accept = false
break
}
}
if accept {
filteredNodes = append(filteredNodes, node)
currentBlock := node.CurrentBlock
if currentBlock >= npool.TopNode.Block {
topNode.Block = currentBlock
topNode.Node = node
}
}
}
return filteredNodes, topNode
} }
// GetNextNode returns next active peer to take a connection // GetNextNode returns next active peer to take a connection
// Loop through entire nodes to find out an alive one and chose one with small CallCounter // Loop through entire nodes to find out an alive one
func GetNextNode(nodes []*Node, topNode TopNodeBlock) *Node { func (bpool *BlockchainPool) GetNextNode(blockchain string) *Node {
nextNode := topNode.Node highestBlock := uint64(0)
for _, node := range nodes { // Get NodePool with correct blockchain
if node.IsAlive() { var np *NodePool
currentBlock := node.CurrentBlock for _, b := range bpool.Blockchains {
if currentBlock < topNode.Block-NB_HIGHEST_BLOCK_SHIFT { if b.Blockchain == blockchain {
// Bypass too outdated nodes np = b
continue for _, n := range b.Nodes {
} if n.CurrentBlock > highestBlock {
if node.CallCounter < nextNode.CallCounter { highestBlock = n.CurrentBlock
nextNode = node }
} }
} }
} }
if nextNode != nil { // Increase Current value with 1
// Increase CallCounter value with 1 currentInc := atomic.AddUint64(&np.Current, uint64(1))
atomic.AddUint64(&nextNode.CallCounter, uint64(1))
}
return nextNode // next is an Atomic incrementer, value always in range from 0 to slice length,
// it returns an index of slice
next := int(currentInc % uint64(len(np.Nodes)))
// Start from next one and move full cycle
l := len(np.Nodes) + next
for i := next; i < l; i++ {
// Take an index by modding with length
idx := i % len(np.Nodes)
// If we have an alive one, use it and store if its not the original one
if np.Nodes[idx].IsAlive() {
if i != next {
// Mark the current one
atomic.StoreUint64(&np.Current, uint64(idx))
}
// Pass nodes with low blocks
// TODO(kompotkot): Re-write to not rotate through not highest blocks
if np.Nodes[idx].CurrentBlock < highestBlock {
continue
}
return np.Nodes[idx]
}
}
return nil
} }
// SetNodeStatus modify status of the node // SetNodeStatus modify status of the node
func SetNodeStatus(url *url.URL, alive bool) { func (bpool *BlockchainPool) SetNodeStatus(url *url.URL, alive bool) {
for _, nodes := range blockchainPools { for _, b := range bpool.Blockchains {
for _, n := range nodes.NodesSet { for _, n := range b.Nodes {
if n.Endpoint.String() == url.String() { if n.Endpoint.String() == url.String() {
n.SetAlive(alive) n.SetAlive(alive)
break break
@ -185,55 +180,55 @@ func SetNodeStatus(url *url.URL, alive bool) {
// StatusLog logs node status // StatusLog logs node status
// TODO(kompotkot): Print list of alive and dead nodes // TODO(kompotkot): Print list of alive and dead nodes
func StatusLog() { func (bpool *BlockchainPool) StatusLog() {
for blockchain, nodes := range blockchainPools { for _, b := range bpool.Blockchains {
for _, n := range nodes.NodesSet { for _, n := range b.Nodes {
log.Printf( log.Printf(
"Blockchain %s node %s is alive %t", "Blockchain %s node %s is alive %t. Blockchain called %d times",
blockchain, n.Endpoint.Host, n.Alive, b.Blockchain, n.Endpoint.Host, n.Alive, b.Current,
) )
} }
} }
} }
// HealthCheck fetch the node latest block // HealthCheck fetch the node latest block
func HealthCheck() { func (bpool *BlockchainPool) HealthCheck() {
for blockchain, nodes := range blockchainPools { for _, b := range bpool.Blockchains {
for _, node := range nodes.NodesSet { for _, n := range b.Nodes {
alive := false alive := false
httpClient := http.Client{Timeout: NB_HEALTH_CHECK_CALL_TIMEOUT} httpClient := http.Client{Timeout: NB_HEALTH_CHECK_CALL_TIMEOUT}
resp, err := httpClient.Post( resp, err := httpClient.Post(
node.Endpoint.String(), n.Endpoint.String(),
"application/json", "application/json",
bytes.NewBuffer([]byte(`{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}`)), bytes.NewBuffer([]byte(`{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}`)),
) )
if err != nil { if err != nil {
node.UpdateNodeState(0, alive) n.UpdateNodeState(0, alive)
log.Printf("Unable to reach node: %s", node.Endpoint.Host) log.Printf("Unable to reach node: %s", n.Endpoint.Host)
continue continue
} }
defer resp.Body.Close() defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body) body, err := ioutil.ReadAll(resp.Body)
if err != nil { if err != nil {
node.UpdateNodeState(0, alive) n.UpdateNodeState(0, alive)
log.Printf("Unable to parse response from %s node, err %v", node.Endpoint.Host, err) log.Printf("Unable to parse response from %s node, err %v", n.Endpoint.Host, err)
continue continue
} }
var statusResponse NodeStatusResponse var statusResponse NodeStatusResponse
err = json.Unmarshal(body, &statusResponse) err = json.Unmarshal(body, &statusResponse)
if err != nil { if err != nil {
node.UpdateNodeState(0, alive) n.UpdateNodeState(0, alive)
log.Printf("Unable to read json response from %s node, err: %v", node.Endpoint.Host, err) log.Printf("Unable to read json response from %s node, err: %v", n.Endpoint.Host, err)
continue continue
} }
blockNumberHex := strings.Replace(statusResponse.Result.Number, "0x", "", -1) blockNumberHex := strings.Replace(statusResponse.Result.Number, "0x", "", -1)
blockNumber, err := strconv.ParseUint(blockNumberHex, 16, 64) blockNumber, err := strconv.ParseUint(blockNumberHex, 16, 64)
if err != nil { if err != nil {
node.UpdateNodeState(0, alive) n.UpdateNodeState(0, alive)
log.Printf("Unable to parse block number from hex to string, err: %v", err) log.Printf("Unable to parse block number from hex to string, err: %v", err)
continue continue
} }
@ -242,24 +237,10 @@ func HealthCheck() {
if blockNumber != 0 { if blockNumber != 0 {
alive = true alive = true
} }
callCounter := node.UpdateNodeState(blockNumber, alive) callCounter := n.UpdateNodeState(blockNumber, alive)
if blockNumber > nodes.TopNode.Block {
nodes.TopNode.Block = blockNumber
nodes.TopNode.Node = node
}
if node.CallCounter >= NB_MAX_COUNTER_NUMBER {
log.Printf(
"Number of CallCounter for node %s reached %d limit, reset the counter.",
node.Endpoint, NB_MAX_COUNTER_NUMBER,
)
atomic.StoreUint64(&node.CallCounter, uint64(0))
}
log.Printf( log.Printf(
"Blockchain %s node %s is alive: %t with current block: %d called: %d times", "Node %s is alive: %t with current block: %d called: %d times", n.Endpoint.Host, alive, blockNumber, callCounter,
blockchain, node.Endpoint.Host, alive, blockNumber, callCounter,
) )
} }
} }

2
nodes/node_balancer/cmd/nodebalancer/cli.go
Wyświetl plik

@ -167,7 +167,7 @@ func (s *StateCLI) populateCLI() {
// Common flag pointers // Common flag pointers
for _, fs := range []*flag.FlagSet{s.addAccessCmd, s.generateConfigCmd, s.deleteAccessCmd, s.serverCmd, s.usersCmd, s.versionCmd} { for _, fs := range []*flag.FlagSet{s.addAccessCmd, s.generateConfigCmd, s.deleteAccessCmd, s.serverCmd, s.usersCmd, s.versionCmd} {
fs.BoolVar(&s.helpFlag, "help", false, "Show help message") fs.BoolVar(&s.helpFlag, "help", false, "Show help message")
fs.StringVar(&s.configPathFlag, "config", "", "Path to configuration file (default: ~/.nodebalancer/config.json)") fs.StringVar(&s.configPathFlag, "config", "", "Path to configuration file (default: ~/.nodebalancer/config.txt)")
} }
// Add, delete and list user access subcommand flag pointers // Add, delete and list user access subcommand flag pointers

40
nodes/node_balancer/cmd/nodebalancer/clients_test.go
Wyświetl plik

@ -1,3 +1,4 @@
// TODO(kompotkot): Re-write tests for client
package main package main
import ( import (
@ -6,36 +7,19 @@ import (
"time" "time"
) )
func setupSuit(t *testing.T) func(t *testing.T) {
t.Log("Setup suit")
configBlockchains = map[string]bool{"ethereum": true}
return func(t *testing.T) {
t.Log("Teardown suit")
}
}
// TestAddClientNode tests adding new client to client pool
func TestAddClientNode(t *testing.T) { func TestAddClientNode(t *testing.T) {
teardownSuit := setupSuit(t)
defer teardownSuit(t)
var cases = []struct { var cases = []struct {
clients map[string]*Client clients map[string]*Client
expected string expected string
}{ }{
{map[string]*Client{"1": {Node: &Node{Alive: true}}}, "1"}, {map[string]*Client{"1": {Node: &Node{Alive: true}}}, "1"},
} }
for _, c := range cases { for _, c := range cases {
CreateClientPools() CreateClientPools()
cpool := GetClientPool("ethereum")
for id, client := range c.clients { for id, client := range c.clients {
cpool.AddClientNode(id, client.Node) ethereumClientPool.AddClientNode(id, client.Node)
} }
for id := range cpool.Client { for id := range ethereumClientPool.Client {
if id != c.expected { if id != c.expected {
t.Log("Wrong client was added") t.Log("Wrong client was added")
t.Fatal() t.Fatal()
@ -44,7 +28,6 @@ func TestAddClientNode(t *testing.T) {
} }
} }
// TestGetClientNode tests getting correct client
func TestGetClientNode(t *testing.T) { func TestGetClientNode(t *testing.T) {
ts := time.Now().Unix() ts := time.Now().Unix()
@ -56,17 +39,15 @@ func TestGetClientNode(t *testing.T) {
{map[string]*Client{}, "1", nil}, {map[string]*Client{}, "1", nil},
{map[string]*Client{"1": {LastCallTs: ts, Node: &Node{Alive: true}}}, "1", &Node{Alive: true}}, {map[string]*Client{"1": {LastCallTs: ts, Node: &Node{Alive: true}}}, "1", &Node{Alive: true}},
{map[string]*Client{"2": {LastCallTs: ts, Node: &Node{Alive: true}}}, "1", nil}, {map[string]*Client{"2": {LastCallTs: ts, Node: &Node{Alive: true}}}, "1", nil},
{map[string]*Client{"1": {LastCallTs: ts - NB_CLIENT_NODE_KEEP_ALIVE, Node: &Node{Alive: true}}}, "1", nil},
} }
for _, c := range cases { for _, c := range cases {
CreateClientPools() CreateClientPools()
cpool := GetClientPool("ethereum")
for id, client := range c.clients { for id, client := range c.clients {
cpool.AddClientNode(id, client.Node) ethereumClientPool.Client[id] = client
} }
clientNode := cpool.GetClientNode(c.id) clientNode := ethereumClientPool.GetClientNode(c.id)
if !reflect.DeepEqual(clientNode, c.expected) { if !reflect.DeepEqual(clientNode, c.expected) {
t.Log("Wrong node returned") t.Log("Wrong node returned")
t.Fatal() t.Fatal()
@ -74,7 +55,6 @@ func TestGetClientNode(t *testing.T) {
} }
} }
// TestCleanInactiveClientNodes tests cleaning inactive clients
func TestCleanInactiveClientNodes(t *testing.T) { func TestCleanInactiveClientNodes(t *testing.T) {
ts := time.Now().Unix() ts := time.Now().Unix()
@ -92,14 +72,12 @@ func TestCleanInactiveClientNodes(t *testing.T) {
} }
for _, c := range cases { for _, c := range cases {
CreateClientPools() CreateClientPools()
cpool := GetClientPool("ethereum")
for id, client := range c.clients { for id, client := range c.clients {
cpool.Client[id] = client ethereumClientPool.Client[id] = client
} }
cpool.CleanInactiveClientNodes() ethereumClientPool.CleanInactiveClientNodes()
for id := range cpool.Client { for id := range ethereumClientPool.Client {
if id != c.expected { if id != c.expected {
t.Log("Wrong client was removed") t.Log("Wrong client was removed")
t.Fatal() t.Fatal()

13
nodes/node_balancer/cmd/nodebalancer/configs.go
Wyświetl plik

@ -18,6 +18,7 @@ var (
nodeConfigs []NodeConfig nodeConfigs []NodeConfig
// Bugout and application configuration // Bugout and application configuration
BUGOUT_AUTH_URL = os.Getenv("BUGOUT_AUTH_URL")
BUGOUT_AUTH_CALL_TIMEOUT = time.Second * 5 BUGOUT_AUTH_CALL_TIMEOUT = time.Second * 5
NB_APPLICATION_ID = os.Getenv("NB_APPLICATION_ID") NB_APPLICATION_ID = os.Getenv("NB_APPLICATION_ID")
NB_CONTROLLER_TOKEN = os.Getenv("NB_CONTROLLER_TOKEN") NB_CONTROLLER_TOKEN = os.Getenv("NB_CONTROLLER_TOKEN")
@ -34,8 +35,7 @@ var (
NB_MAX_COUNTER_NUMBER = uint64(10000000) NB_MAX_COUNTER_NUMBER = uint64(10000000)
// Client configuration // Client configuration
NB_CLIENT_NODE_KEEP_ALIVE = int64(5) // How long to store node in hot list for client in seconds NB_CLIENT_NODE_KEEP_ALIVE = int64(5) // How long to store node in hot list for client in seconds
NB_HIGHEST_BLOCK_SHIFT = uint64(50) // Allowed shift to prefer node with most highest block
NB_ACCESS_ID_HEADER = os.Getenv("NB_ACCESS_ID_HEADER") NB_ACCESS_ID_HEADER = os.Getenv("NB_ACCESS_ID_HEADER")
NB_DATA_SOURCE_HEADER = os.Getenv("NB_DATA_SOURCE_HEADER") NB_DATA_SOURCE_HEADER = os.Getenv("NB_DATA_SOURCE_HEADER")
@ -60,9 +60,8 @@ func CheckEnvVarSet() {
// Nodes configuration // Nodes configuration
type NodeConfig struct { type NodeConfig struct {
Blockchain string `json:"blockchain"` Blockchain string `json:"blockchain"`
Endpoint string `json:"endpoint"` Endpoint string `json:"endpoint"`
Tags []string `json:"tags"`
} }
func LoadConfig(configPath string) error { func LoadConfig(configPath string) error {
@ -109,7 +108,7 @@ func GetConfigPath(providedPath string) (*ConfigPlacement, error) {
return nil, fmt.Errorf("Unable to find user home directory, %v", err) return nil, fmt.Errorf("Unable to find user home directory, %v", err)
} }
configDirPath = fmt.Sprintf("%s/.nodebalancer", homeDir) configDirPath = fmt.Sprintf("%s/.nodebalancer", homeDir)
configPath = fmt.Sprintf("%s/config.json", configDirPath) configPath = fmt.Sprintf("%s/config.txt", configDirPath)
} else { } else {
configPath = strings.TrimSuffix(providedPath, "/") configPath = strings.TrimSuffix(providedPath, "/")
configDirPath = filepath.Dir(configPath) configDirPath = filepath.Dir(configPath)
@ -145,7 +144,7 @@ func GenerateDefaultConfig(config *ConfigPlacement) error {
if !config.ConfigExists { if !config.ConfigExists {
tempConfig := []NodeConfig{ tempConfig := []NodeConfig{
{Blockchain: "ethereum", Endpoint: "http://127.0.0.1:8545", Tags: []string{"local"}}, {Blockchain: "ethereum", Endpoint: "http://127.0.0.1:8545"},
} }
tempConfigJson, err := json.Marshal(tempConfig) tempConfigJson, err := json.Marshal(tempConfig)
if err != nil { if err != nil {

14
nodes/node_balancer/cmd/nodebalancer/middleware.go
Wyświetl plik

@ -98,14 +98,14 @@ func (ac *AccessCache) Cleanup() (int64, int64) {
return removedAccessIds, totalAccessIds return removedAccessIds, totalAccessIds
} }
func initCacheCleaning() { func initCacheCleaning(debug bool) {
t := time.NewTicker(NB_CACHE_CLEANING_INTERVAL) t := time.NewTicker(NB_CACHE_CLEANING_INTERVAL)
for { for {
select { select {
case <-t.C: case <-t.C:
removedAccessIds, totalAccessIds := accessIdCache.Cleanup() removedAccessIds, totalAccessIds := accessIdCache.Cleanup()
if stateCLI.enableDebugFlag { if debug {
log.Printf("[DEBUG] Removed %d elements from access id cache", removedAccessIds) log.Printf("Removed %d elements from access id cache", removedAccessIds)
} }
log.Printf("Elements in access id cache: %d", totalAccessIds) log.Printf("Elements in access id cache: %d", totalAccessIds)
} }
@ -241,7 +241,7 @@ func logMiddleware(next http.Handler) http.Handler {
if stateCLI.enableDebugFlag { if stateCLI.enableDebugFlag {
if r.URL.RawQuery != "" { if r.URL.RawQuery != "" {
logStr += fmt.Sprintf(" [DEBUG] %s", r.URL.RawQuery) logStr += fmt.Sprintf(" %s", r.URL.RawQuery)
} }
accessID := extractAccessID(r) accessID := extractAccessID(r)
if accessID != "" { if accessID != "" {
@ -269,20 +269,20 @@ func accessMiddleware(next http.Handler) http.Handler {
// If access id does not belong to internal crawlers, then check cache or find it in Bugout resources // If access id does not belong to internal crawlers, then check cache or find it in Bugout resources
if accessID == NB_CONTROLLER_ACCESS_ID { if accessID == NB_CONTROLLER_ACCESS_ID {
if stateCLI.enableDebugFlag { if stateCLI.enableDebugFlag {
log.Printf("[DEBUG] Access id belongs to internal crawlers") log.Printf("Access id belongs to internal crawlers")
} }
currentClientAccess = internalCrawlersAccess currentClientAccess = internalCrawlersAccess
currentClientAccess.dataSource = dataSource currentClientAccess.dataSource = dataSource
} else if accessIdCache.FindAccessIdInCache(accessID) != "" { } else if accessIdCache.FindAccessIdInCache(accessID) != "" {
if stateCLI.enableDebugFlag { if stateCLI.enableDebugFlag {
log.Printf("[DEBUG] Access id found in cache") log.Printf("Access id found in cache")
} }
currentClientAccess = accessIdCache.accessIds[accessID] currentClientAccess = accessIdCache.accessIds[accessID]
currentClientAccess.dataSource = dataSource currentClientAccess.dataSource = dataSource
accessIdCache.UpdateAccessIdAtCache(accessID, dataSource) accessIdCache.UpdateAccessIdAtCache(accessID, dataSource)
} else { } else {
if stateCLI.enableDebugFlag { if stateCLI.enableDebugFlag {
log.Printf("[DEBUG] New access id, looking at Brood resources") log.Printf("New access id, looking at Brood resources")
} }
resources, err := bugoutClient.Brood.GetResources( resources, err := bugoutClient.Brood.GetResources(
NB_CONTROLLER_TOKEN, NB_CONTROLLER_TOKEN,

56
nodes/node_balancer/cmd/nodebalancer/routes.go
Wyświetl plik

@ -53,12 +53,25 @@ func lbHandler(w http.ResponseWriter, r *http.Request) {
return return
} }
// Chose one node
var node *Node
cpool := GetClientPool(blockchain)
node = cpool.GetClientNode(currentClientAccess.AccessID)
if node == nil {
node = blockchainPool.GetNextNode(blockchain)
if node == nil {
http.Error(w, "There are no nodes available", http.StatusServiceUnavailable)
return
}
cpool.AddClientNode(currentClientAccess.AccessID, node)
}
// Save origin path, to use in proxyErrorHandler if node will not response // Save origin path, to use in proxyErrorHandler if node will not response
r.Header.Add("X-Origin-Path", r.URL.Path) r.Header.Add("X-Origin-Path", r.URL.Path)
switch { switch {
case strings.HasPrefix(r.URL.Path, fmt.Sprintf("/nb/%s/jsonrpc", blockchain)): case strings.HasPrefix(r.URL.Path, fmt.Sprintf("/nb/%s/jsonrpc", blockchain)):
lbJSONRPCHandler(w, r, blockchain, currentClientAccess) lbJSONRPCHandler(w, r, blockchain, node, currentClientAccess)
return return
default: default:
http.Error(w, fmt.Sprintf("Unacceptable path for %s blockchain %s", blockchain, r.URL.Path), http.StatusBadRequest) http.Error(w, fmt.Sprintf("Unacceptable path for %s blockchain %s", blockchain, r.URL.Path), http.StatusBadRequest)
@ -66,7 +79,7 @@ func lbHandler(w http.ResponseWriter, r *http.Request) {
} }
} }
func lbJSONRPCHandler(w http.ResponseWriter, r *http.Request, blockchain string, currentClientAccess ClientResourceData) { func lbJSONRPCHandler(w http.ResponseWriter, r *http.Request, blockchain string, node *Node, currentClientAccess ClientResourceData) {
body, err := ioutil.ReadAll(r.Body) body, err := ioutil.ReadAll(r.Body)
if err != nil { if err != nil {
http.Error(w, "Unable to read body", http.StatusBadRequest) http.Error(w, "Unable to read body", http.StatusBadRequest)
@ -81,43 +94,6 @@ func lbJSONRPCHandler(w http.ResponseWriter, r *http.Request, blockchain string,
return return
} }
// Get tags from request params, sort and generate from it identifier
var tags []string
queries := r.URL.Query()
for k, v := range queries {
if k == "tag" {
for _, tag := range v {
tags = append(tags, tag)
}
}
}
// Chose one node
var node *Node
cpool := GetClientPool(blockchain)
node = cpool.GetClientNode(currentClientAccess.AccessID)
if node == nil {
npool := blockchainPools[blockchain]
var nodes []*Node
var topNode TopNodeBlock
if len(tags) != 0 {
nodes, topNode = npool.FilterTagsNodes(tags)
} else {
topNode = npool.TopNode
nodes = npool.NodesSet
}
node = GetNextNode(nodes, topNode)
if node == nil {
http.Error(w, "There are no nodes available", http.StatusServiceUnavailable)
return
}
cpool.AddClientNode(currentClientAccess.AccessID, node)
}
if stateCLI.enableDebugFlag {
log.Printf("[DEBUG] Used node with endpoint: %s, call counter equals: %d", node.Endpoint, node.CallCounter)
}
switch { switch {
case currentClientAccess.dataSource == "blockchain": case currentClientAccess.dataSource == "blockchain":
if currentClientAccess.BlockchainAccess == false { if currentClientAccess.BlockchainAccess == false {
@ -134,6 +110,8 @@ func lbJSONRPCHandler(w http.ResponseWriter, r *http.Request, blockchain string,
} }
} }
node.IncreaseCallCounter()
// Overwrite Path so response will be returned to correct place // Overwrite Path so response will be returned to correct place
r.URL.Path = "/" r.URL.Path = "/"
node.GethReverseProxy.ServeHTTP(w, r) node.GethReverseProxy.ServeHTTP(w, r)

70
nodes/node_balancer/cmd/nodebalancer/server.go
Wyświetl plik

@ -5,6 +5,7 @@ package main
import ( import (
"context" "context"
"encoding/json"
"fmt" "fmt"
"log" "log"
"net/http" "net/http"
@ -28,12 +29,12 @@ var (
) )
// initHealthCheck runs a routine for check status of the nodes every 5 seconds // initHealthCheck runs a routine for check status of the nodes every 5 seconds
func initHealthCheck() { func initHealthCheck(debug bool) {
t := time.NewTicker(NB_HEALTH_CHECK_INTERVAL) t := time.NewTicker(NB_HEALTH_CHECK_INTERVAL)
for { for {
select { select {
case <-t.C: case <-t.C:
HealthCheck() blockchainPool.HealthCheck()
logStr := "Client pool healthcheck." logStr := "Client pool healthcheck."
for b := range configBlockchains { for b := range configBlockchains {
cp := clientPool[b] cp := clientPool[b]
@ -41,8 +42,8 @@ func initHealthCheck() {
logStr += fmt.Sprintf(" Active %s clients: %d.", b, clients) logStr += fmt.Sprintf(" Active %s clients: %d.", b, clients)
} }
log.Println(logStr) log.Println(logStr)
if stateCLI.enableDebugFlag { if debug {
StatusLog() blockchainPool.StatusLog()
} }
} }
} }
@ -88,7 +89,7 @@ func proxyErrorHandler(proxy *httputil.ReverseProxy, url *url.URL) {
} }
// After 3 retries, mark this backend as down // After 3 retries, mark this backend as down
SetNodeStatus(url, false) blockchainPool.SetNodeStatus(url, false)
// Set modified path back // Set modified path back
// TODO(kompotkot): Try r.RequestURI instead of header // TODO(kompotkot): Try r.RequestURI instead of header
@ -128,24 +129,33 @@ func Server() {
fmt.Printf("Unable to get user with provided access identifier, err: %v\n", err) fmt.Printf("Unable to get user with provided access identifier, err: %v\n", err)
os.Exit(1) os.Exit(1)
} }
resourcesLog := "Access with resources established."
if len(resources.Resources) != 1 { if len(resources.Resources) != 1 {
log.Printf("%s There are no access IDs for users in resources", resourcesLog) fmt.Printf("User with provided access identifier has wrong number of resources, err: %v\n", err)
} else { os.Exit(1)
log.Printf("%s Found user access IDs in resources", resourcesLog) }
resource_data, err := json.Marshal(resources.Resources[0].ResourceData)
if err != nil {
fmt.Printf("Unable to encode resource data interface to json, err: %v\n", err)
os.Exit(1)
}
var clientAccess ClientResourceData
err = json.Unmarshal(resource_data, &clientAccess)
if err != nil {
fmt.Printf("Unable to decode resource data json to structure, err: %v\n", err)
os.Exit(1)
} }
// Set internal crawlers access to bypass requests from internal services
// without fetching data from authn Brood server
internalCrawlersUserID := uuid.New().String()
internalCrawlersAccess = ClientResourceData{ internalCrawlersAccess = ClientResourceData{
UserID: internalCrawlersUserID, UserID: clientAccess.UserID,
AccessID: NB_CONTROLLER_ACCESS_ID, AccessID: clientAccess.AccessID,
Name: "InternalCrawlersAccess", Name: clientAccess.Name,
Description: "Access for internal crawlers.", Description: clientAccess.Description,
BlockchainAccess: true, BlockchainAccess: clientAccess.BlockchainAccess,
ExtendedMethods: true, ExtendedMethods: clientAccess.ExtendedMethods,
} }
log.Printf(
"Internal crawlers access set, resource id: %s, blockchain access: %t, extended methods: %t",
resources.Resources[0].Id, clientAccess.BlockchainAccess, clientAccess.ExtendedMethods,
)
err = InitDatabaseClient() err = InitDatabaseClient()
if err != nil { if err != nil {
@ -185,18 +195,14 @@ func Server() {
r.Header.Del(strings.Title(NB_DATA_SOURCE_HEADER)) r.Header.Del(strings.Title(NB_DATA_SOURCE_HEADER))
// Change r.Host from nodebalancer's to end host so TLS check will be passed // Change r.Host from nodebalancer's to end host so TLS check will be passed
r.Host = r.URL.Host r.Host = r.URL.Host
// Explicit set of r.URL requires, because by default it adds trailing slash and brake some urls
r.URL = endpoint
} }
proxyErrorHandler(proxyToEndpoint, endpoint) proxyErrorHandler(proxyToEndpoint, endpoint)
newNode := &Node{ blockchainPool.AddNode(&Node{
Endpoint: endpoint, Endpoint: endpoint,
Alive: true, Alive: true,
GethReverseProxy: proxyToEndpoint, GethReverseProxy: proxyToEndpoint,
} }, nodeConfig.Blockchain)
AddNode(nodeConfig.Blockchain, nodeConfig.Tags, newNode)
log.Printf( log.Printf(
"Added new %s proxy blockchain under index %d from config file with geth url: %s://%s", "Added new %s proxy blockchain under index %d from config file with geth url: %s://%s",
nodeConfig.Blockchain, i, endpoint.Scheme, endpoint.Host) nodeConfig.Blockchain, i, endpoint.Scheme, endpoint.Host)
@ -205,12 +211,6 @@ func Server() {
// Generate map of clients // Generate map of clients
CreateClientPools() CreateClientPools()
// Start node health checking and current block fetching
HealthCheck()
if stateCLI.enableHealthCheckFlag {
go initHealthCheck()
}
serveMux := http.NewServeMux() serveMux := http.NewServeMux()
serveMux.Handle("/nb/", accessMiddleware(http.HandlerFunc(lbHandler))) serveMux.Handle("/nb/", accessMiddleware(http.HandlerFunc(lbHandler)))
log.Println("Authentication middleware enabled") log.Println("Authentication middleware enabled")
@ -227,8 +227,14 @@ func Server() {
WriteTimeout: 40 * time.Second, WriteTimeout: 40 * time.Second,
} }
// Start node health checking and current block fetching
blockchainPool.HealthCheck()
if stateCLI.enableHealthCheckFlag {
go initHealthCheck(stateCLI.enableDebugFlag)
}
// Start access id cache cleaning // Start access id cache cleaning
go initCacheCleaning() go initCacheCleaning(stateCLI.enableDebugFlag)
log.Printf("Starting node load balancer HTTP server at %s:%s", stateCLI.listeningAddrFlag, stateCLI.listeningPortFlag) log.Printf("Starting node load balancer HTTP server at %s:%s", stateCLI.listeningAddrFlag, stateCLI.listeningPortFlag)
err = server.ListenAndServe() err = server.ListenAndServe()

2
nodes/node_balancer/cmd/nodebalancer/version.go
Wyświetl plik

@ -1,3 +1,3 @@
package main package main
var NB_VERSION = "0.2.2" var NB_VERSION = "0.2.1"

2
nodes/node_balancer/sample.env
Wyświetl plik

@ -1,5 +1,5 @@
# Required environment variables for load balancer # Required environment variables for load balancer
export BUGOUT_BROOD_URL="https://auth.bugout.dev" export BUGOUT_AUTH_URL="https://auth.bugout.dev"
export NB_APPLICATION_ID="<application_id_to_controll_access>" export NB_APPLICATION_ID="<application_id_to_controll_access>"
export NB_CONTROLLER_TOKEN="<token_of_controller_user>" export NB_CONTROLLER_TOKEN="<token_of_controller_user>"
export NB_CONTROLLER_ACCESS_ID="<controller_access_id_for_internal_crawlers>" export NB_CONTROLLER_ACCESS_ID="<controller_access_id_for_internal_crawlers>"