It has been said that the most valuable resource in the 21st century is no longer oil but data. With most of the data being controlled by 6 or 7 tech companies many people have started developing peer to peer/ decentralized systems again to try and take back control of data and mitigate the risk of relying on cloud providers. In this article I’ll explore the use of the Kademlia DHT and the Iota tangle for storing data on a decentralized network of nodes.
Software Libraries used
Overview
Kademlia is a distributed hash table. It uses key-value pairs similarly to regular hash maps but it stores them on various nodes connected to it. It searches for the node with an ID that closely resembles the key in the routing table of each node until it finds the closest match. This allows it to find and store data in O(log(n)) time while only needing a small amount of addresses in the routing table. I used the Kadence library to implement a Kademlia network.
Iota is a distributed ledger technology similar to blockchain but has a completely different structure. It is incredibly fast and is useful for projects where you use micro transactions.
A basic Kademlia node will be ran that allows us to store and retrieve small bits of data and the Iota token will be used to pay small amounts to any node that stores the data.
Project Discussion
Kademlia requires two or more nodes to store data on the network. In my code I set up two nodes running on localhost and used the join method so one would connect to the other.
const node = new kadence.KademliaNode({
identity: kadence.utils.getRandomKeyBuffer(),
transport: new kadence.UDPTransport(),
storage: levelup(encode(leveldown('database'))),
contact: { hostname: 'localhost', port: 1337 }
});
//Creating second node object//
const nodeTwo = new kadence.KademliaNode({
identity: kadence.utils.getRandomKeyBuffer(),
transport: new kadence.UDPTransport(),
storage: levelup(encode(leveldown('secondDB'))),
contact: { hostname: 'localhost', port: 8080 }
});
//Listening on ports of each node//
node.listen(1337);
nodeTwo.listen(8080);
//Node joins network through nodeTwo//
node.join([ nodeTwo.identity, {
hostname: 'localhost',
port: 8080
}], async () => {
//Rest of the article's code
//goes in here besides the use
//Methods and the sendIota function
});
The next thing required is to use the iterativeStore() function. This takes a key and a value as parameters. It finds K closest node IDs to the key and stores them on that node’s local storage.
//Details for key and Value we want to store//
const key = kadence.utils.getRandomKeyBuffer();
const Value = 'Elon Did Nothing Wrong'
console.log('The key is: ');
console.log(key);
node.iterativeStore(key, Value, function (err, data) {
if (err) {
return console.error(err);
}
else{
console.log(data);
}
});
The KadenceNode class also has a function called iterativeFindValue. It normally returns an array of Entities which are a type with a Node ID, value and timestamp. This is what is needed to return the value again but for some reason when you first call the function it just returns an Array of Node IDs that the value was stored on. Happy accident since I can use this to send Iota tokens to all the nodes. I made a handler for each of the nodes that would send their Iota address. I then created a function called sendIota() that would connect to an Iota full node, prepare a transaction then attach it to the tangle.
node.use('IOTA', (req, res) => {
const iota = 'QVFEHDRITCVU9LTPSNQGGQRELGSSBOMQG9XFWSLBRTUYDZMUFLSOWGYNS9ZW9UOAYWSBBJKHFZESDGWMZOQKDQOUSY';
res.send(iota);
});
nodeTwo.use('IOTA', (req, res) => {
const iota = 'CEEMLNFZSOOXVCTDNXZXWZBVVYZTJW9XKBWPHXKABB9FCSJMNLXZJ9UZTODXBSFSGRCYOILEFSQSTREPY9EGVFZTHZ';
res.send(iota);
});
function sendIota(Address, Seed){
const iota = iotaCore.composeAPI({ provider: 'https://testnet140.tangle.works'})
const msg = converter.asciiToTrytes('Thanks for the storage');
// Array of transfers which defines transfer recipients and value transferred in IOTAs.
const transfers = [{
address: Address,
value: 0, // 1Ki
tag: '', // optional tag of 0-27 trytes
message: msg // optional message in trytes
}]
iota.prepareTransfers(Seed, transfers)
.then(trytes => iota.sendTrytes(trytes, 3, 14))
.then(bundle => {
console.log(`Published transaction with tail hash: ${bundle[0].hash}`)
console.log(`Bundle: ${bundle}`)
res.send(bundle[0].hash)
})
.catch(err => {
// catch any errors
})
}
The last thing I needed to do was search through the array of Node IDs returned by the iterativeFindValue() function and call the sendIota() function for each address. I did this using a for loop.
node.iterativeFindValue(key, function (error, value, contact){
if (error) {
return console.error(err);
}
else{
const resp = value;
//const publisher = value.publisher;
const len = resp.length;
for(var i = 0; i < len; i++){
node.send('IOTA', ['Iota Address'], resp[i], function (err, result){
if (error) {
return console.error(err);
}
else{
console.log(result);
sendIota(result, mySeed);
}
});
}
}
});
The code creates two nodes and stores data across each of them. It then sends Iota tokens to the address of any node that stores the data. This is a very broad overview of how DHTs and Iota could be used for storing files and is not meant to actually be used. My code can be found in my github below along with my social media.
Thanks for reading.