added offset block list getter

README.md

@@ -21,7 +21,7 @@
 | [Docs](#documentation)/[web copy](https://beardog108.github.io/onionr/)               | [Get involved](#help-out)  | [Onionr.net](https://onionr.net/)/[.onion](http://onionrbak72t5zhbzuey2fdkpczlvhowgcpqc6uoyrd3uxztzxwz5cyd.onion/)                  |
 
 
-<hr>
+---
 
 **The main repository for this software is at https://git.VoidNet.tech/kev/onionr/**
 
@@ -30,20 +30,26 @@ Mirrors: [Github](https://github.com/beardog108/onionr), [Gitlab](https://gitlab
 
 Onionr ("Onion Relay") is a decentralized/distributed peer-to-peer communication network, designed to be anonymous and resistant to (meta)data analysis, spam, and corruption.
 
-Onionr stores data in independent packages referred to as 'blocks'. The blocks are distributed to all interested nodes. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor and as such cannot be trivially unmasked. Anonymity is achieved by a stateless network, with no given indication of what node a block originates from. In fact, since one is not required to participate in routing or storage to insert a message, blocks often do not originate from any identifiable node.
+Onionr gives the individual the ability to speak freely, without fear of surveillance and censorship.
+
+---
+
+Onionr stores data in independent packages referred to as 'blocks'. The blocks are distributed to all nodes interested in their data type. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor and as such cannot be trivially unmasked. Anonymity is achieved by a stateless network, with no given indication of what node a block originates from. In fact, since one is not required to participate in routing or storage to insert a message, blocks often do not originate from any identifiable node.
+
+Onionr works primarily via epidemic/gossip style routing, with message delivery taking roughly log<sub>F</sub>(N) cycles where F is the number of nodes to send a message to each cycle and N is the number of connected nodes. So a network of 100 million nodes can deliver messages in a few minutes even with high packet loss and malfunctioning nodes.
 
 Through message mixing and key privacy, it is intended to be nigh impossible to discover the identity of a message creator or recipient. Via long-term traffic analysis, a well funded adversary may discover the most probable node(s) to be creating a set of related blocks, however doing so would only lead them to a node behind Tor. As the first node that a block appears on is almost always not the creator of the block, there is plausible deniability regarding the true creator of the block.
 
-Onionr gives the individual the ability to speak freely, without fear of surveillance and censorship.
-
 Users are identified by ed25519/curve25519 public keys, which can be used to sign blocks or send encrypted data.
 
 Onionr can be used for mail, as a social network, instant messenger, file sharing software, or for encrypted group discussion.
 
-Due to the nature of anonymity, the graph as implemented in this reference network is dense, undirected, cyclic and can be disconnected. As a result, current scalability is poor but sufficient for high latency communications. As the need arises isolated stream solutions may be implemented (in a manner similar to described in the Bitmessage whitepaper). Since Onionr is technically just a data format, any routing scheme can be used to pass messages.
+Due to the nature of anonymity, the graph as implemented in this reference network is dense, undirected, cyclic and can be disconnected. Since Onionr is technically just a data format, any routing scheme can be used to pass messages.
 
 The whitepaper is available [here](docs/whitepaper.md).
 
+---
+
 ## Main Features
 
 * [X] 🌐 Fully p2p/decentralized, no trackers or other single points of failure
@@ -57,9 +63,9 @@ Onionr ships with various application plugins ready for use out of the box:
 
 Currently usable:
 
-* Mail
+* 📨 Mail
-* Public anonymous chat/message board
+* 💬 Public anonymous chat/message board
-* Simple webpage hosting - Will be greatly extended
+* 📃 Simple webpage hosting - Will be greatly extended
 * File sharing (Work in progress)
 
 Not yet usable:
@@ -156,8 +162,6 @@ Donating at least $3 gets you cool Onionr stickers. Get in touch if you want the
 
 * USD (Card/Paypal (no account required)): [Ko-Fi](https://www.ko-fi.com/beardogkf)
 
-* [Indiegogo](https://igg.me/at/onionr/x#/)
-
 * Sign up for [privacy.com (refferal link)](https://privacy.com/join/FNNDF) to protect your personal information when contributing or shopping elsewhere, we both get $5 USD.
 
 Note: probably not tax deductible

docs/dev/selected-papers.md

@@ -7,3 +7,4 @@ A paper being listed here is not end-all-be-all endorsement of every detail insi
 * [Protecting Free Expression Online with Freenet](https://freenetproject.org/assets/papers/ddisrs.pdf)
 * [Bitmessage: A Peer‐to‐Peer Message Authentication and Delivery System](https://archive.org/details/BitmessageWhitepaper/)
 * [MuON: Epidemic based Mutual Anonymity](https://web.archive.org/web/20060901153544/http://www.csl.mtu.edu/cs6461/www/Reading/MuON_ICNP2005.pdf)
+* [SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol](https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf)

static-data/default-plugins/torgossip/commandhandlers.py

@@ -2,6 +2,7 @@
 
 Handle commands for the torgossip server
 """
+from typing import type_check_only
 from onionrblocks import generators
 from onionrblocks.generators import anonvdf
 import blockio
@@ -22,7 +23,10 @@ GNU General Public License for more details.
 You should have received a copy of the GNU General Public License
 along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
+
+
 def put_block(safe_db, block):
+    #6
     block_hash = block[:64]
     data = block[64:]
     try:
@@ -36,19 +40,31 @@ def put_block(safe_db, block):
     return b"1"
 
 
-
 def get_block(safe_db, block_hash) -> bytes:
-    # 4
+    #5
     try:
         return safe_db.get(block_hash)
     except KeyError:
         return b"0"
 
 
+def list_blocks_by_type_and_offset(safe_db, type_and_offset):
+    #4
+    offset, block_type = type_and_offset.split(b',', 1)
+    try:
+        offset = int(offset)
+    except ValueError:
+        return b""
+    try:
+        return list_blocks_by_type(safe_db, block_type)[offset * 64:]
+    except KeyError:
+        return b"0"
+
+
 def list_blocks_by_type(safe_db, block_type) -> bytes:
     # 3
     block_type = block_type.decode('utf-8')
-    print('ty', block_type)
+
     try:
         return safe_db.get('bl-' + block_type)
     except KeyError:

static-data/default-plugins/torgossip/commands.py

@@ -24,6 +24,7 @@ class GossipCommands(IntEnum):
     PING = 1,
     CHECK_HAS_BLOCK = 2,
     LIST_BLOCKS_BY_TYPE = 3,
-    GET_BLOCK = 4,
+    LIST_BLOCKS_BY_TYPE_OFFSET = 4,
-    PUT_BLOCK = 5,
+    GET_BLOCK = 5,
-    EXIT = 6
+    PUT_BLOCK = 6,
+    EXIT = 7

static-data/default-plugins/torgossip/runtest.py

@@ -18,6 +18,12 @@ def torgossip_runtest(test_manager):
     bl = subprocgenerate.vdf_block(b"test", "txt", 100)
     blockio.store_block(bl, test_manager._too_many.get_by_string("SafeDB"))
 
+    tsts = b''
+    for i in range(3):
+        bl = subprocgenerate.vdf_block(b"test" + os.urandom(3), "tst", 100)
+        tsts += bl.id
+        blockio.store_block(bl, test_manager._too_many.get_by_string("SafeDB"))
+
     bl_new = blockcreator.create_anonvdf_block(b"test5", "txt", 10)
 
 
@@ -34,21 +40,28 @@ def torgossip_runtest(test_manager):
         assert bl.id in s.recv(10000)
 
         # test getting a block that doesn't exist
-        s.sendall(b'4' + os.urandom(64))
+        s.sendall(b'5' + os.urandom(64))
         assert s.recv(64) == b"0"
 
         # test getting a block that does exist
-        s.sendall(b'4' + bl.id)
+        s.sendall(b'5' + bl.id)
         assert s.recv(64) == bl.get_packed()
 
-        s.sendall(b'5' + bl_new.id + bl_new.get_packed())
+        # test putting block
+        s.sendall(b'6' + bl_new.id + bl_new.get_packed())
         assert s.recv(2) == b"1"
 
         # test block was uploaded by getting it
-        s.sendall(b'4' + bl_new.id)
+        s.sendall(b'5' + bl_new.id)
         assert s.recv(64) == bl_new.get_packed()
 
         # test block was uploaded by getting it
-        s.sendall(b'6')
+        s.sendall(b'7')
         assert s.recv(64) == b"BYE"
 
+        s.sendall(b'41tst')
+        assert s.recv(1000) == tsts[64:]
+
+        s.sendall(b'42tst')
+        assert s.recv(1000) == tsts[64*2:]
+

static-data/default-plugins/torgossip/server_test.py

@@ -1,29 +0,0 @@
-#!/usr/bin/env python
-
-import socket
-import os
-
-#home = input("Enter Onionr data directory")
-home = "/dev/shm/DATA24688"
-
-def client(data):
-    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
-        s.connect(f'{home}/torgossip.sock')
-        s.sendall(data)
-        resp = s.recv(32)
-        print("\n", resp)
-
-while True:
-    print("1. ping")
-    print('2. check block hash')
-    print('3 list blocks')
-    print("4. exit")
-    inp = input()
-    if inp == "1":
-        client(b'1')
-    elif inp == "3":
-        client(b"3" + input("type: ").encode('utf8'))
-    elif inp == "2":
-        client(b'2' + os.urandom(32))
-    elif inp == "3":
-        client(b'3')