Using Hive Blockchain Custom Json Operation with Python & Beem

avatar

hivewp20.pngImage by @doze

One of Hive blockchain's powerful features is its custom_json operation. Normally, users don't even see and hear about this operations. However, this operation enables developers to store their game or web application data on the Hive blockchain. If you would like to know more about what is stored in Hive blocks and and curious about various Hive operations, feel free to read my Exploring Hive Blocks post.

In this post I would like to share how to broadcast custom_json operations to Hive blockchain, how to retrieve them afterwards using Beem module in Python. Beem is a python module created by @holger80, and makes is easy to connect to Hive blockchain and interact with it using python programming language.

Custom_json is one of many operations of Hive. It allow developers to store game or applications data on Hive blockchain. Data is stored as text in Json format. Sine Json is a native JavaScript format and widely used in website and web applications, Hive makes it super easy to create Apps and Games that would like to use decentralized solutions.

Broadcasting a Transaction to Hive Blockchain

Every Hive block has a size limit of 65536 bytes. Which means we can only store 65536 characters including all the properties in one block. I believe this is a parameters that witnesses can change if they wish to based on demand, infrastructure, scaling, etc. For now and foreseeable future this is the limitation every developer of apps and games should consider.

Another limitation is each custom_json operation is limited to 8192 bytes or characters. However, it is possible to broadcast multiple operations within a transaction. We face another limitation there. We can only broadcast 5 operations within a transaction by the broadcasting account.

Before we get to the code, another thing to keep in mind is, once transaction is built and we broadcast it, the return value doesn't not give the block number where the transaction is stored. That is because when we broadcast with python we don't wait until the block is produced. A couple minutes after broadcasting the transaction the block is produced and available for us to retrieve the data stored. When we broadcast we get back values like expiration, ref_block_num and ref_block_prefix. Using these three values we can find our transaction and the block it is stored in. So that we don't search the entire blockchain form the beginning, I also decided to store the current block number before broadcasting. That way I can start searching for the transaction starting from that block number.

Following is the code to broadcast custom_json operations:

from beem.transactionbuilder import TransactionBuilder
from beembase.operations import Custom_json
from beem import Hive
from beem.blockchain import Blockchain 
from beem.block import Block
from helper import wif, name 
from beem.nodelist import NodeList
from pprint import pprint

nodelist = NodeList()
nodelist.update_nodes()
nodes = nodelist.get_hive_nodes()

hive = Hive(node=nodes, nobroadcast=False, keys={'posting': wif})
blockchain = Blockchain(hive)
current_block_num = blockchain.get_current_block_num()

tx = TransactionBuilder(blockchain_instance=hive)

for i in range(1,6):
  txt = str(i) * 8100
  data = {}
  data['data'] = txt
  cj = {
      "required_auths": [],
      "required_posting_auths": [name],
      "id": "librarian",
      "json": data
    }
  tx.appendOps(Custom_json(cj))

tx.appendWif(wif)
signed_tx = tx.sign()
broadcast_tx = tx.broadcast()

print(current_block_num)
print(broadcast_tx)

Main thing to pay attention to is the TransactionBuilder. It allows us to build a transaction before broadcasting then broadcast it. I stored Hive account name and keys in a separate python file named helper.py, and imported variables name and wif to sign the transaction. To broadcast a this transaction posting key was enough.

We can add upto 5 operations in our transaction using .appendOps() method. Everything else should be self-explanatory. Feel free to ask in the comments if something doesn't make sense. I will be happy to explain each line.

Now the last two print statements show the block number before the transaction was broadcasted, and return values after the broadcast happened.

Now we need to retrieve the data from Hive. We will use the printed or stored block number to start the search from and values returned when we broadcasted the transactions such as expiration, ref_block_num, and ref_block_prefix.

from beem import Hive
from beem.block import Block
from pprint import pprint
from beem.blockchain import Blockchain
from beem.nodelist import NodeList

nodelist = NodeList()
nodelist.update_nodes()
nodes = nodelist.get_hive_nodes()
hive = Hive(node=nodes)

start_block_num = 52374262
blockchain = Blockchain(hive)
end_block_num = blockchain.get_current_block_num()

exp = '2021-03-23T02:08:41'
rbn = 10998
rbp = 3231133325

def get_data(start, end, exp, rbn, rbp):
    ref = []
    for num in range(start, end):
        block = Block(block=num, blockchain_instance=hive)
        b = block.json()
        for i, tr in enumerate(b['transactions']):
            if tr['expiration'] == exp and tr['ref_block_num'] == rbn and tr['ref_block_prefix'] == rbp:
                print(i)
                print(b['id'])
                print(b['timestamp'])
                print(tr['operations'])
                print(b['transaction_ids'][i])
                ref.append(b['id'])
                ref.append(i)
                return ref

ref = get_data(start_block_num, end_block_num, exp, rbn, rbp)

print('-----')
print(ref)
bb = Block(block=ref[0])
bb = bb.json()
bb = bb['transactions'][ref[1]]
bb = bb['operations'][0]['value']['json']

print(bb)

What the code above does is, it searches for the stored transaction with custom_json operations starting with the nearest known block numbers. Once found it displays the data we are searching for. It also provides the block number the transaction is stored at.

Now, we want to store this block number as reference for future use. Next time we need the same data, we can just get the block from the blockchain instead of iterating through multiple blocks and save us time.

If we have the block number, then we can easily get everything stored in that block as following:

block_num = 52374284

block = Block(block=block_num, blockchain_instance=hive)
b = block.json()

pprint(b)

One last thing I would like to share has to do with resource credits. I used an account with 1006 HP to test the codes above. The transaction above had 5 operations with 8100 characters each in size. It only took less than 1% of resource credits for this account to broadcast the transaction.

I hope you find this post useful. Let me know your thoughts. Also, let me know if there are better ways of broadcasting transactions and retrieving data from Hive blockchain using python.

Posted Using LeoFinance Beta



0
0
0.000
20 comments
avatar

You really are so knowledgeable about this matter. Before I was wondering what was Json means but time flies by staying on the chain. I learned about the command but I don't know how this command became possible. Well, I'm not in tech things but I still do enjoy this kind of stuff. Thanks

Posted Using LeoFinance Beta

0
0
0.000
avatar

Thank you. I actually started learning python here because of my interest how Hive works. It is not that difficult and I know only the basics.

If you are interested in programming at all, I recommend looking into python. It is easy and simple and anybody can learn it in no time.

Posted Using LeoFinance Beta

0
0
0.000
avatar

But I do not know programming so maybe it's impossible for me. hehehe

0
0
0.000
avatar

This.

You're like me.. Dove into coding more just to mess around on here. :)

0
0
0.000
avatar

Congratulations @geekgirl! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

You received more than 90000 upvotes.
Your next target is to reach 95000 upvotes.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

0
0
0.000
avatar

oh super cool. I'm learning python at the moment, this will help. Want to plot some stats on hive. any starting tips?

0
0
0.000
avatar

For stats on Hive I would suggest also using HiveSQL. It is free. Getting data is faster with HiveSQL.

Maybe focus on one thing and try on both: Beem and HiveSQL, then see which one performs faster.

That way you will be able to do the task using both, but use the fastest one.

I am also planning to work on Wallet related stats, like how much money is going in and out of exchanges.

If you are going to use Beem, make sure to read their documentation and use the parts that help with your project. (https://beem.readthedocs.io/en/latest/)

If you choose to use HiveSQL, you can also use SQL commands within python scripts.

If you decide to use HiveSQL within python scripts, you may find this post useful - https://hive.blog/hive-167922/@geekgirl/hivesql-in-python-which-module-to-choose-pypyodbc-pyodbc-or-pymssql.

Feel free to ask anytime you have any questions.

Posted Using LeoFinance Beta

0
0
0.000
avatar

thanks a bunch, will dig into it

0
0
0.000
avatar

I'm using something like this in my code perhaps others might like this.

I have an option so that during testing, instead of spamming the chain I'm creating transaction files in a folder. I'm getting the secret stuff (I'm also only using the Posting key on this) from Environment variables which I'm setting in VSCode at home and in Heroku's configuration settings for the cloud version.

from beem import Hive

def send_custom_json(custom_json, id = 'v4v_consolidate'):
    """ Send a custom_json to Hive or write to a transaction file return trx_id """
    post_to_hive = current_app.config['HIVE_POST_TRANSACTIONS']
    if post_to_hive:
        try:
            # Now send the custom JSON
            wif = [current_app.config['HIVE_WALLET_PRIVATE_KEY']]
            server_account = current_app.config['HIVE_SERVER_ACCOUNT']
            h = Hive(keys=wif)
            tx = h.custom_json(id=id, json_data= custom_json ,
                            required_posting_auths=[server_account])
            trx_id = tx['trx_id']
            current_app.logger.info(f'Json saved in https://hive.ausbit.dev/tx/{trx_id}')

        except Exception as ex:
            current_app.logger.warning(f'{ex.__class__} occurred')
            trx_id = "failure"
    else:
        # Don't post to HIVE.
        # Use this SHA_1 Hash for testing if we don't want to write to the chain.
        sha_1 = hashlib.sha1(json.dumps(custom_json, default=str).encode()).hexdigest()
        trx_id = sha_1
        current_app.logger.info(f'New TX - {trx_id}')
        if not os.path.isdir('trx'):
            os.makedirs('trx')
        with open(os.path.join('trx', str(trx_id) + '.json'), 'w') as f:
            json.dump(custom_json, f, indent=2, default=str)

    return trx_id


0
0
0.000
avatar

Awesome. Thank you. This does help.

How do you set environment variables in VSCode? I had to find a certain file to make that happen. Already forgot how I did that. lol. Any easy way for setting env variables locally?

Posted Using LeoFinance Beta

0
0
0.000
avatar

I have always wanted to learn python, but it has been so long since I coded that I feel like I am too old now. I was never that good at it anyway. Nice post and I hope that those who are not coding adverse like me find a wealth of knowledge here!

Posted Using LeoFinance Beta

0
0
0.000
avatar

Thank you very much. One of the main reasons I like python is its ability to automate repetitive tasks. It is worth learning for those who have repetitive tasks that can be automated to save time. Otherwise it is boring. :)

Thank You.

Posted Using LeoFinance Beta

0
0
0.000
avatar

Is it similar to any other language? I used to use C++ when I was in college, but like I said, that was a long long time ago!

Posted Using LeoFinance Beta

0
0
0.000
avatar

Yes it shares common data types, operators, logic with other languages. Syntax is different. It is a general purpose language. It is a lot easier and simpler than other languages, in my opinion. More human readable. A lot easier than C++.

Posted Using LeoFinance Beta

0
0
0.000
avatar

Ah, I might have to find some online courses and tinker around with it. I know that practice makes perfect, so if I put some time into it I might be able to do better than I did in college!

Posted Using LeoFinance Beta

0
0
0.000
avatar
(Edited)

Not exactly the info I was looking for at this moment, but this is very good information.

Just a heads up.... your post is #1 result for "custom+json+hive+blockchain" on Google... It was for me.

!LUV the information, thanks again.

Posted Using LeoFinance Beta

0
0
0.000