The Big, Ugly, Nasty Update

ctgpp.png

It was bound to happen...

Things were pretty smooth. A hiccup every now and again, but generally, we've been enjoying smooth sailing on the blockchain for the past few years of building on it.

Then...Last week happened!

We'll go over a few of the issues, some fixes that are in place and how we are going to grow and learn from this but let's start with the elephant in the room:

The Hive Engine dev team released an update for light nodes and domains (v 1.7.2 to 1.7.3), so we decided to be proactive and get our servers up to date with the changes. Things didn't work out that smoothly. And for the better part of the weekend up until a few hours ago, our stuff basically went offline.

Blain has been working around the clock while in the middle of a family move to try to get things sorted and as soon as we found the fix, other witness servers started mentioning there were issues. Hey let's just say we are ahead of the game...But the damage was done.

ListNerds Was Offline

This was a glaring issue that most people noticed when LISTNERDS rewards stopped being sent. There are a few days of rewards that should be sent out tonight or tomorrow and things will be back to normal soon. The good news, we were able to do work with the server to improve deliverability and speed of the queue. But still, the site just wasn't working properly over the weekend.

HivePay Swap Had Issues

This is the biggest issue. Because of the issues with our server, the SWAP feature of HivePay basically stopped working and people sent SWAP but did not get refunded immediately. And then once our server got back online, it actually double paid the refunds.

Which drained HivePay Swap's liquidity. We are sorting through all the data, but from what we can tell, from around block # 65413973 SWAP basically double refunded customers and the pool was drained. It's our bad, and we will make sure we get liquidity back, but we are going to tag everyone that SWAP refunded over the past few hours and will ask for anyone that may have been double refunded to send the SWAP or Hive back to @hivepayswap

We know this isn't ideal and we do apologize for this, but if we can recoup a chunk of the liquidity it will help Hive Pay Swap get back to normal and functioning again.

As for future issues, Blain has found it and put the fix in place so the double refunds don't happen again. Regardless of what happens, the CTP Team will make sure liquidity is back in the SWAP!

Apologies in advance for tagging so many people, but we just want to make sure you check to see if the funds got sent in error:

@hariton
@lanny0938
@aiovo
@forsakensushi
@gadrian
@pu77up
@swaps
@savvyplayer
@simhot
@damla
@marketroll
@iamalivechalleng
@uyobong
@ordet
@cryptokungfu
@colskr
@vxc
@matiflesh
@burlarj
@roger.remix
@diecinueve
@pixelfan
@simhot
@globetrottergcc
@mango-juice
@vxxxc
@eddiespino
@writeandearn
@h3m4n7
@ijat
@birdwatcher
@creodas
@mypathtofire
@matiflesh
@alexicp
@p30505seamew

Again, apologies for tagging everyone, but we hope to see if we can get some help to recover some of these funds.

So all in all....

Growing pains. And a massive inconvenience for our members this past weekend. Which we cannot apologize enough for. We are confident the fixes in place will make sure any updates by Hive Engine in the future will not hurt our services and platforms.

Thank you so much for your patience and understanding and we hope to put this behind us and continue to grow on the blockchain!

Jon, Blain and Eliana
The CTP SWARM

P.S. If you have any questions at all, or need clarification, be sure to join our Discord server here! We are also transitioning all our Telegram communications into Discord going forward, which will help us serve our community much better!



0
0
0.000
59 comments
avatar

You all rock. Here is some !PIZZA for being awesome and building a great community.

0
0
0.000
avatar

Thanks bro, rough weekend, but we'll get through it!

0
0
0.000
avatar

Heard about these issues from another h-e witness.

0
0
0.000
avatar

Yeah man, just a messed up weekend. But we'll get it all sorted :)

0
0
0.000
avatar

Growing pains. We'd like to think we have run everything flawlessly, ALL the time... but it just doesn't tend to happen that way. Sometimes we simply have to accept that 99% right, 99% of the time is actually a pretty good score.

I, for one, am very grateful for the SWAP utility.

=^..^=

0
0
0.000
avatar

We had a pretty good track record of being online for the past few years. lol Not a bad run!!!

0
0
0.000
avatar

Now that you're back up and running, this has the chance to be a really cool case study...

We know that alongside the balances in people's wallets, Hive accounts also have a social value tied to them based on their owner's actions and standing within the community.

While Hive's immutable accounts and decentralised network mean you can't force those that got double refunds paid into their account to send them back, irreversible damage could be done to their reputation if they choose not to.

So those users are now faced with 2 choices:

  1. Send the money back, thus keeping their social standing and ongoing earning potential via having their content supported with upvotes from the community high.
  2. Keep the money, giving them a short term windfall but potentially branding them as shit people and leading to no ongoing support.

Yeah, science!

Posted Using LeoFinance Beta

0
0
0.000
avatar

That's a really good point. It actually happened once before but to one customer. They sent it back right away and completely understood the mix up. It'll be interesting to see how many send it back once they are aware.

Crossing fingers :)

0
0
0.000
avatar

Glad you got it fixed and are back and I hope the folks that got money back twice will man up and send the second payment back to you. All the best

0
0
0.000
avatar

I dont have any doubts that the Hive community will work with us :)

0
0
0.000
avatar

Thank you for the update :) You must have worked tirelessly. Hoping things will work smoothly after the fixes.

Posted Using LeoFinance Beta

0
0
0.000
avatar

Blain hasnt slept. He will make sure everything is 100%

0
0
0.000
avatar

Sorry to hear, hope he is okay despite not having slept.
Salute to you all for all the hard work you do behind the scenes :)

Posted Using LeoFinance Beta

0
0
0.000
avatar

Hello it looks like I was sent some hive as a result of this, can you check to see if what came to me was correct or not, I have used hive pay swap so many times, and I often dont even check to see if it came correctly. I will hold onto what I have now until you get back to me, its about 190 hive.

!PIZZA

!LUV

!LOLZ

!PGM

0
0
0.000
avatar

Sent 0.1 PGM - 0.1 LVL- 1 STARBITS - 0.05 DEC - 15 SBT tokens to @forsakensushi, @ctptalk

remaining commands 9

BUY AND STAKE THE PGM TO SEND A LOT OF TOKENS!

The tokens that the command sends are: 0.1 PGM-0.1 LVL-2.5 BUDS-0.01 MOTA-0.05 DEC-15 SBT-1 STARBITS-[0.00000001 BTC (SWAP.BTC) only if you have 2500 PGM in stake or more ]

5000 PGM IN STAKE = 2x rewards!

image.png
Discord image.png

Support the curation account @ pgm-curator with a delegation 10 HP - 50 HP - 100 HP - 500 HP - 1000 HP

Get potential votes from @ pgm-curator by paying in PGM, here is a guide

I'm a bot, if you want a hand ask @ zottone444


0
0
0.000
avatar
(Edited)

Thanks for the hard work during such inconvenient times in real life. I don't know if it's an option for Listnerds as it grows much bigger, but I know top Hive witnesses have backup nodes to avoid such situations when they upgrade, or if one node goes down. Of course, these are extra costs for redundancy you will hardly ever need.

Posted using LeoFinance Mobile

PS: Returned 115.937 HIVE to @hivepayswap.

image.png

image.png

0
0
0.000
avatar

I was kind of busy so I didn't realize there were so many issues. I hope you get back your liquidity for Hivepay back.

Posted Using LeoFinance Beta

0
0
0.000
avatar

Sorry about the issues guys. I sent 536 Hive to @hivepayswap back. Hope you'll get all your funds back soon. Feel free to reach out if there's anything else I can do.

resim


resim

0
0
0.000
avatar

Thanks for the update and the hard work behind the scene to get things back on track.

0
0
0.000
avatar

Good time to check it out again and see if I can make use of it!

0
0
0.000
avatar

Hi guys,
I just send back the 5 hive I received by mistake.
But...I don't understand why I received it in the first place because...I didn't use your service for weeks. (as far as I remember). So maybe your problem(bug) is larger than a double refund...

0
0
0.000
avatar

We have found the issue and are implementing updates to prevent it from happening again

0
0
0.000
avatar

Great to hear. So, next time you send me HIVE, I can keep it 😉😂
Joking of course. Keep up the good work 👍

0
0
0.000
avatar

Sorry about the issues, I saw I was refunded Hive and Swap.Hive. I made the refunds for both below. Let me know if there is anything else. Thanks for all your work guys, I know situations like this are tough.

image

image

image

0
0
0.000
avatar

Growing pains indeed....Wishing you the best of luck

0
0
0.000
avatar

I will check mine and hit you on discord

0
0
0.000
avatar

Wow. That is a Big, Ugly, Nasty Update indeed. Glad you solved it and let's hope it will never happen again. I hope you'll manage to recover the funds.

onwards and upwards

0
0
0.000
avatar

Just to update on this. HivePayswap is currently not running until @blainjones finalizes the updates and pushes it out. Please do not try to swap at this time. I will push another update as a reply here as well as in the HivePay and CTP discord servers and the website will be functioning again when it is back up and running

0
0
0.000
avatar

Knew something was up and never batted an eye. @blainjones @elianaicgomes @jongolson have earned and generated that all important trust factor. I wasn't on the list, but it would have been a no-brainer to have given it back. More importantly, thank you for providing us with the all important Big Ugly Nasty Update and not bs'ing us about it.

!CTP

0
0
0.000
avatar

thanks for the update.
Good to hear that the issues related to witness server has been resolved and listnerds and hive is back to normal.
I am sure most of them will turn-up and refund the amounts done wrongly or errorneously by the system.

I was suprised to see messages on ecency of refunds but no transactions or amounts :-)
That may be a glitch in ecency notification system.

Happy week ahead.

0
0
0.000
avatar

Hello! This is harsh to read, especially

image.png

Ouch!

As one of the top hive-engine witnesses (@locolombia), I am interested in figuring out exactly what error occurred and how it could be avoided in the future! Was this an error on the hive-engine side or on the interaction with the hive-engine protocol?

Feel free to reach out to me, if I or my team can do anything to help you guys even an independent code review, we are happy to help. God Bless you and keep building guys!

0
0
0.000
avatar
(Edited)

I feel with you. Why all these things happen at inopportune times and over the weekend is beyond me.

Hopefully, people will send the funds back.

Yes, backup node, hot spare with different upgrade policy,... We can all be smart after the fact.

@tipu curate

0
0
0.000
avatar

These things happen, but I know it was a terrible time for you guys with your moves as well. Oh well, just Murphey's Law.
It is nice to see things looking more normal on @ListNerds this morning.

0
0
0.000
avatar

Hi Jon and Blain. I totally understand and know how it goes. I'm glad it's almost ready for you to put behind. I knew what was going on. Get some rest.
Barb

0
0
0.000
avatar

I have just noticed this mention, but I have already just refunded the balance in Hive that was sent to me by mistake.

0
0
0.000
avatar

By the way, this explains why the service has been disabled during these days, I hope they will soon fix everything and enable it again, this is one of the services I use the most.

0
0
0.000
avatar

Your team are doing a great job. Thanks for all the fixes, effort and help (replies).
Also, I'm very happy about the heart-to-heart talk at the Hive community.
From my side, I'll be patient, open to help and a big thank you.

0
0
0.000