bug fixes and performence improvements

This commit is contained in:
Kevin Froman 2019-01-21 20:26:56 -06:00
parent 10bed5f9c8
commit 4d5e0aeb74
5 changed files with 30 additions and 27 deletions

View File

@ -39,21 +39,21 @@ When designing Onionr we had these main goals in mind:
At its core, Onionr is merely a description for storing data in self-verifying packages ("blocks"). These blocks can be encrypted to a user (or for one's self), encrypted symmetrically, or not at all. Blocks can be signed by their creator, but regardless, they are self-verifying due to being identified by a sha3-256 hash value; once a block is created, it cannot be modified. At its core, Onionr is merely a description for storing data in self-verifying packages ("blocks"). These blocks can be encrypted to a user (or for one's self), encrypted symmetrically, or not at all. Blocks can be signed by their creator, but regardless, they are self-verifying due to being identified by a sha3-256 hash value; once a block is created, it cannot be modified.
Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however this is configurable. Blocks do not rely on any particular order of reciept or transport mechanism. Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however this is configurable. Blocks do not rely on any particular order of receipt or transport mechanism.
## User IDs ## User IDs
User IDs are simply Ed25519 public keys. They are represented in Base32 format, or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list). User IDs are simply Ed25519 public keys. They are represented in Base32 format, or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list).
Public keys can be generated deterministicly with a password using a key derivation function (Argon2id). This password can be shared between many users in order to share data anonymously among a group, using only 1 password. This is useful in some cases, but is risky, as if one user causes the key to be compromised and does not notify the group or revoke the key, there is no way to know. Public keys can be generated deterministically with a password using a key derivation function (Argon2id). This password can be shared between many users in order to share data anonymously among a group, using only 1 password. This is useful in some cases, but is risky, as if one user causes the key to be compromised and does not notify the group or revoke the key, there is no way to know.
## Nodes ## Nodes
Although Onionr is transport agnostic, the only supported transports in the reference implemetation are Tor .onion services and I2P hidden services. Nodes announce their address on creation by connecting to peers specified in a bootstrap file. Peers in the bootstrap file have no special permissions aside from being default peers. Although Onionr is transport agnostic, the only supported transports in the reference implementation are Tor .onion services and I2P hidden services. Nodes announce their address on creation by connecting to peers specified in a bootstrap file. Peers in the bootstrap file have no special permissions aside from being default peers.
### Node Profiling ### Node Profiling
To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the amount of successful block transfers, speed, and reliabilty of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connection to 'friend' nodes. To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the amount of successful block transfers, speed, and reliability of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connection to 'friend' nodes.
## Block Format ## Block Format
@ -90,3 +90,5 @@ This can be done either by the creator of the block prior to generation, or by a
In addition, randomness beacons such as the one operated by [NIST](https://beacon.nist.gov/home) or the hash of the latest blocks in a cryptocurrency network could be used to affirm that a block was at least not *created* before a given time. In addition, randomness beacons such as the one operated by [NIST](https://beacon.nist.gov/home) or the hash of the latest blocks in a cryptocurrency network could be used to affirm that a block was at least not *created* before a given time.
# Direct Connections # Direct Connections
We propose a method of using Onionr's block sync system to enable direct connections between peers by having one peer request to connect to another using the peer's public key. Since the request is within a standard block, proof of work must be used to request connection, and in addition neither party knows the other's .onion or .i2p hostname, let alone IP address. If the requested peer is available and wishes to accept the connection,Onionr will generate a temporary .onion address for the other peer to connect to. Alternatively, a reverse connection may be formed, which is faster to establish but requires message brokering instead of a standard socket.

View File

@ -105,7 +105,7 @@ def check():
open(get_config_file(), 'a', encoding="utf8").close() open(get_config_file(), 'a', encoding="utf8").close()
save() save()
except: except:
logger.warn('Failed to check configuration file.') logger.debug('Failed to check configuration file.')
def save(): def save():
''' '''

View File

@ -130,7 +130,7 @@ class Core:
events.event('pubkey_add', data = {'key': peerID}, onionr = None) events.event('pubkey_add', data = {'key': peerID}, onionr = None)
conn = sqlite3.connect(self.peerDB, timeout=10) conn = sqlite3.connect(self.peerDB, timeout=30)
hashID = self._crypto.pubKeyHashID(peerID) hashID = self._crypto.pubKeyHashID(peerID)
c = conn.cursor() c = conn.cursor()
t = (peerID, name, 'unknown', hashID, 0) t = (peerID, name, 'unknown', hashID, 0)
@ -160,7 +160,7 @@ class Core:
if type(address) is None or len(address) == 0: if type(address) is None or len(address) == 0:
return False return False
if self._utils.validateID(address): if self._utils.validateID(address):
conn = sqlite3.connect(self.addressDB, timeout=10) conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor() c = conn.cursor()
# check if address is in database # check if address is in database
# this is safe to do because the address is validated above, but we strip some chars here too just in case # this is safe to do because the address is validated above, but we strip some chars here too just in case
@ -193,7 +193,7 @@ class Core:
''' '''
if self._utils.validateID(address): if self._utils.validateID(address):
conn = sqlite3.connect(self.addressDB, timeout=10) conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor() c = conn.cursor()
t = (address,) t = (address,)
c.execute('Delete from adders where address=?;', t) c.execute('Delete from adders where address=?;', t)
@ -213,7 +213,7 @@ class Core:
''' '''
if self._utils.validateHash(block): if self._utils.validateHash(block):
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
t = (block,) t = (block,)
c.execute('Delete from hashes where hash=?;', t) c.execute('Delete from hashes where hash=?;', t)
@ -261,7 +261,7 @@ class Core:
raise Exception('Block db does not exist') raise Exception('Block db does not exist')
if self._utils.hasBlock(newHash): if self._utils.hasBlock(newHash):
return return
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
currentTime = self._utils.getEpoch() + self._crypto.secrets.randbelow(301) currentTime = self._utils.getEpoch() + self._crypto.secrets.randbelow(301)
if selfInsert or dataSaved: if selfInsert or dataSaved:
@ -318,7 +318,7 @@ class Core:
#blockFile.write(data) #blockFile.write(data)
#blockFile.close() #blockFile.close()
onionrstorage.store(self, data, blockHash=dataHash) onionrstorage.store(self, data, blockHash=dataHash)
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
c.execute("UPDATE hashes SET dataSaved=1 WHERE hash = ?;", (dataHash,)) c.execute("UPDATE hashes SET dataSaved=1 WHERE hash = ?;", (dataHash,))
conn.commit() conn.commit()
@ -341,7 +341,7 @@ class Core:
if not os.path.exists(self.queueDB): if not os.path.exists(self.queueDB):
self.dbCreate.createDaemonDB() self.dbCreate.createDaemonDB()
else: else:
conn = sqlite3.connect(self.queueDB, timeout=10) conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor() c = conn.cursor()
try: try:
for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'): for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'):
@ -367,7 +367,7 @@ class Core:
retData = True retData = True
date = self._utils.getEpoch() date = self._utils.getEpoch()
conn = sqlite3.connect(self.queueDB, timeout=10) conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor() c = conn.cursor()
t = (command, data, date, responseID) t = (command, data, date, responseID)
@ -410,7 +410,7 @@ class Core:
''' '''
Clear the daemon queue (somewhat dangerous) Clear the daemon queue (somewhat dangerous)
''' '''
conn = sqlite3.connect(self.queueDB, timeout=10) conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor() c = conn.cursor()
try: try:
@ -428,7 +428,7 @@ class Core:
''' '''
Return a list of addresses Return a list of addresses
''' '''
conn = sqlite3.connect(self.addressDB, timeout=10) conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor() c = conn.cursor()
if randomOrder: if randomOrder:
addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();') addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();')
@ -456,7 +456,7 @@ class Core:
randomOrder determines if the list should be in a random order randomOrder determines if the list should be in a random order
trust sets the minimum trust to list trust sets the minimum trust to list
''' '''
conn = sqlite3.connect(self.peerDB, timeout=10) conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor() c = conn.cursor()
payload = '' payload = ''
@ -505,7 +505,7 @@ class Core:
trust int 4 trust int 4
hashID text 5 hashID text 5
''' '''
conn = sqlite3.connect(self.peerDB, timeout=10) conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor() c = conn.cursor()
command = (peer,) command = (peer,)
@ -531,7 +531,7 @@ class Core:
Update a peer for a key Update a peer for a key
''' '''
conn = sqlite3.connect(self.peerDB, timeout=10) conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor() c = conn.cursor()
command = (data, peer) command = (data, peer)
@ -563,7 +563,7 @@ class Core:
introduced 10 introduced 10
''' '''
conn = sqlite3.connect(self.addressDB, timeout=10) conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor() c = conn.cursor()
command = (address,) command = (address,)
@ -588,7 +588,7 @@ class Core:
Update an address for a key Update an address for a key
''' '''
conn = sqlite3.connect(self.addressDB, timeout=10) conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor() c = conn.cursor()
command = (data, address) command = (data, address)
@ -609,7 +609,7 @@ class Core:
if dateRec == None: if dateRec == None:
dateRec = 0 dateRec = 0
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
# if unsaved: # if unsaved:
@ -630,7 +630,7 @@ class Core:
Returns the date a block was received Returns the date a block was received
''' '''
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
execute = 'SELECT dateReceived FROM hashes WHERE hash=?;' execute = 'SELECT dateReceived FROM hashes WHERE hash=?;'
@ -646,7 +646,7 @@ class Core:
Returns a list of blocks by the type Returns a list of blocks by the type
''' '''
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
if orderDate: if orderDate:
@ -665,7 +665,7 @@ class Core:
def getExpiredBlocks(self): def getExpiredBlocks(self):
'''Returns a list of expired blocks''' '''Returns a list of expired blocks'''
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
date = int(self._utils.getEpoch()) date = int(self._utils.getEpoch())
@ -683,7 +683,7 @@ class Core:
Sets the type of block Sets the type of block
''' '''
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
c.execute("UPDATE hashes SET dataType = ? WHERE hash = ?;", (blockType, hash)) c.execute("UPDATE hashes SET dataType = ? WHERE hash = ?;", (blockType, hash))
conn.commit() conn.commit()
@ -710,7 +710,7 @@ class Core:
if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'): if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'):
return False return False
conn = sqlite3.connect(self.blockDB, timeout=10) conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor() c = conn.cursor()
args = (data, hash) args = (data, hash)
c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args) c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args)

View File

@ -0,0 +1 @@
svlegnabtuh3dq6ncmzqmpnxzik5mita5x22up4tai2ekngzcgqbnbqd.onion

View File

@ -79,7 +79,7 @@
"peers" : { "peers" : {
"minimum_score" : -100, "minimum_score" : -100,
"max_stored_peers" : 5000, "max_stored_peers" : 5000,
"max_connect" : 10 "max_connect" : 50
}, },
"timers" : { "timers" : {