work on new peer profiling system

docs/api.md

@@ -1,34 +1,2 @@
-BLOCK HEADERS (simple ID system to identify block type)
------------------------------------------------
--crypt- (encrypted block)
--bin- (binary file)
--txt- (plaintext)
-
 HTTP API
-------------------------------------------------
-/client/ (Private info, not publicly accessible)
-
--   hello
-    - hello world
--   shutdown
-    - exit onionr
--   stats
-    - show node stats
-
-/public/
-
--   firstConnect
-    - initialize with peer
--   ping
-    - pong
--   setHMAC
-    - set a created symmetric key 
--   getDBHash
-    - get the hash of the current hash database state
--   getPGP
-    - export node's PGP public key
--   getData
-    - get a data block
--   getBlockHashes
-    - get a list of the node's hashes
--------------------------------------------------
+TODO

docs/onionr-draft.md

@@ -1,57 +0,0 @@
-# Onionr Protocol Spec v2
-
-A P2P platform for Tor & I2P
-
-# Overview
-
-Onionr is an encrypted microblogging & mailing system designed in the spirit of Twitter.
-There are no central servers and all traffic is peer to peer by default (routed via Tor or I2P).
-User IDs are simply Tor onion service/I2P host id + Ed25519 key fingerprint.
-Private blocks are only able to be read by the intended peer.
-All traffic is over Tor/I2P, connecting only to Tor onion and I2P hidden services.
-
-## Goals:
-    • Selective sharing of information
-    • Secure & semi-anonymous direct messaging
-    • Forward secrecy
-    • Defense in depth
-    • Data should be secure for years to come
-    • Decentralization
-    * Avoid browser-based exploits that plague similar software
-    * Avoid timing attacks & unexpected metadata leaks
-
-## Protocol
-
-Onionr nodes use HTTP (over Tor/I2P) to exchange keys, metadata, and blocks. Blocks are identified by their sha3_256 hash. Nodes sync a table of blocks hashes and attempt to download blocks they do not yet have from random peers.
-
-Blocks may be encrypted using Curve25519 or Salsa20.
-
-Blocks have IDs in the following format:
-
--Optional hash of public key of publisher (base64)-optional signature (non-optional if publisher is specified) (Base64)-block type-block hash(sha3-256)
-
-pubkeyHash-signature-type-hash
-
-## Connections
-
-When a node first comes online, it attempts to bootstrap using a default list provided by a client.
-When two peers connect, they exchange Ed25519 keys (if applicable) then Salsa20 keys.
-
-Salsa20 keys are regenerated either every X many communications with a peer or every X minutes. 
-
-Every 100kb or every 2 hours is a recommended default.
-
-All valid requests with HMAC should be recorded until used HMAC's expiry to prevent replay attacks.
-Peer Types
-    * Friends:
-        * Encrypted ‘friends only’ posts to one another
-        * Usually less strict rate & storage limits
-    * Strangers:
-        * Used for storage of encrypted or public information
-        * Can only read public posts
-        * Usually stricter rate & storage limits
-
-## Spam mitigation
-
-To send or receive data, a node can optionally request that the other node generate a hash that when in hexadecimal representation contains a random string at a random location in the string. Clients will configure what difficulty to request, and what difficulty is acceptable for themselves to perform. Difficulty should correlate with recent network & disk usage and data size. Friends can be configured to have less strict (to non existent) limits, separately from strangers. (proof of work).
-Rate limits can be strict, as Onionr is not intended to be an instant messaging application.

onionr/communicator2.py

@@ -20,7 +20,7 @@
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 import sys, os, core, config, json, onionrblockapi as block, requests, time, logger, threading, onionrplugins as plugins, base64, onionr
-import onionrexceptions
+import onionrexceptions, onionrpeers
 from defusedxml import minidom
 
 class OnionrCommunicatorDaemon:
@@ -48,6 +48,7 @@ class OnionrCommunicatorDaemon:
         # lists of connected peers and peers we know we can't reach currently
         self.onlinePeers = []
         self.offlinePeers = []
+        self.peerProfiles = [] # list of peer's profiles (onionrpeers.PeerProfile instances)
 
         # amount of threads running by name, used to prevent too many
         self.threadCounts = {}
@@ -262,6 +263,7 @@ class OnionrCommunicatorDaemon:
         '''Adds a new random online peer to self.onlinePeers'''
         retData = False
         tried = self.offlinePeers
+        peerScores = {}
         if peer != '':
             if self._core._utils.validateID(peer):
                 peerList = [peer]
@@ -274,6 +276,14 @@ class OnionrCommunicatorDaemon:
             # Avoid duplicating bootstrap addresses in peerList
             self.addBootstrapListToPeerList(peerList)
 
+        for address in peerList:
+            # Load peer's profiles into a list
+            profile = onionrpeers.PeerProfiles(address, self._core)
+            peerScores[address] = profile.score
+
+        # Sort peers by their score, greatest to least
+        peerList = sorted(peerScores, key=peerScores.get, reverse=True)
+
         for address in peerList:
             if len(address) == 0 or address in tried or address in self.onlinePeers:
                 continue
@@ -299,7 +309,7 @@ class OnionrCommunicatorDaemon:
                 logger.info(i)
 
     def peerAction(self, peer, action, data=''):
-        '''Perform a get request to a peer'''
+        '''Perform a get request to a peer'''        
         if len(peer) == 0:
             return False
         logger.info('Performing ' + action + ' with ' + peer + ' on port ' + str(self.proxyPort))
@@ -348,6 +358,8 @@ class OnionrCommunicatorDaemon:
         self.decrementThreadCount('daemonCommands')
 
     def uploadBlock(self):
+        '''Upload our block to a few peers'''
+        # when inserting a block, we try to upload it to a few peers to add some deniability
         triedPeers = []
         if not self._core._utils.validateHash(self.blockToUpload):
             logger.warn('Requested to upload invalid block')

onionr/core.py

@@ -575,9 +575,10 @@ class Core:
         # TODO: validate key on whitelist
         if key not in ('address', 'type', 'knownPeer', 'speed', 'success', 'DBHash', 'failure', 'lastConnect'):
             raise Exception("Got invalid database key when setting address info")
-        c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
-        conn.commit()
-        conn.close()
+        else:
+            c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
+            conn.commit()
+            conn.close()
         return
 
     def getBlockList(self, unsaved = False): # TODO: Use unsaved??

onionr/onionrpeers.py

@@ -1,7 +1,7 @@
 '''
     Onionr - P2P Microblogging Platform & Social network.
 
-    This file contains both the OnionrCommunicate class for communcating with peers
+    This file contains both the PeerProfiles class for network profiling of Onionr nodes
 '''
 '''
     This program is free software: you can redistribute it and/or modify
@@ -16,4 +16,57 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+'''
+import core
+class PeerProfiles:
+    '''
+        PeerProfiles
+    '''
+    def __init__(self, address, coreInst):
+        self.address = address # node address
+        self.score = None
+        self.friendSigCount = 0
+        self.success = 0
+        self.failure = 0
+
+        if not isinstance(coreInst, core.Core):
+            raise TypeError("coreInst must be a type of core.Core")
+        self.coreInst = coreInst
+        assert isinstance(self.coreInst, core.Core)
+
+        self.loadScore()
+        return
+
+    def loadScore(self):
+        '''Load the node's score from the database'''
+        try:
+            self.success = int(self.coreInst.getAddressInfo('success'))
+        except TypeError:
+            self.success = 0
+        try:
+            self.failure = int(self.coreInst.getAddressInfo('failure'))
+        except TypeError:
+            self.failure = 0
+        self.score = self.success - self.failure
+    
+    def saveScore(self):
+        '''Save the node's score to the database'''
+        self.coreInst.setAddressInfo(self.address, 'success', self.success)
+        self.coreInst.setAddressInfo(self.address, 'failure', self.failure)
+        return
+
+def getScoreSortedPeerList(coreInst):
+    if not type(coreInst is core.Core):
+        raise TypeError('coreInst must be instance of core.Core')
+
+    peerList = coreInst.listAdders()
+    peerScores = {}
+
+    for address in peerList:
+        # Load peer's profiles into a list
+        profile = PeerProfiles(address, coreInst)
+        peerScores[address] = profile.score
+
+    # Sort peers by their score, greatest to least
+    peerList = sorted(peerScores, key=peerScores.get, reverse=True)
+    return peerList

onionr/static-data/default_config.json

@@ -37,5 +37,8 @@
         "netTotal": 1000000000,
         "blockCache" : 5000000,
         "blockCacheTotal" : 50000000
+    },
+    "peers":{
+        "minimumScore": 5
     }
 }